CompletableFuture๋ฅผ ํ™œ์šฉํ•œ ์„ฑ๋Šฅ ๊ฐœ์„ ๊ธฐ

2024. 8. 1. 22:54ยทProject

๋“ค์–ด๊ฐ€๊ธฐ ์•ž์„œ

์ €ํฌ ํ”„๋กœ์ ํŠธ์—์„œ๋Š” ๋งค์ผ ์˜ค์ „ 4์‹œ์— ๋ฒ ์ŠคํŠธ ์…€๋Ÿฌ ๋„์„œ 100๊ถŒ์˜ ์ •๋ณด๋ฅผ ํฌ๋กค๋งํ•˜์—ฌ ์„œ๋ฒ„์— ์ €์žฅํ•˜๋Š” ์ž‘์—…์„ ์ˆ˜ํ–‰ํ•ด์•ผ ํ–ˆ์Šต๋‹ˆ๋‹ค. ์ฒ˜์Œ์—๋Š” ๋™๊ธฐ์ ์ธ ๋ฐฉ์‹์œผ๋กœ ์ฝ”๋“œ๋ฅผ ์ž‘์„ฑํ•˜์—ฌ ์ด ์ž‘์—…์„ ํ…Œ์ŠคํŠธํ•ด ๋ณด์•˜์œผ๋‚˜, ์ด ๋ฐฉ๋ฒ•์œผ๋กœ๋Š” ๋ฐ์ดํ„ฐ ํฌ๋กค๋ง์— ๋งŽ์€ ์‹œ๊ฐ„์ด ์†Œ์š”๋˜์—ˆ๊ณ , ์ „์ฒด ํ”„๋กœ์„ธ์Šค ํšจ์œจ์„ฑ์— ๋ฌธ์ œ๊ฐ€ ์žˆ์—ˆ์Šต๋‹ˆ๋‹ค. 

 

์ด์— ๋”ฐ๋ผ, ์ด ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด CompletableFuture๋ฅผ ์ด์šฉํ•œ ๋น„๋™๊ธฐ ์ฒ˜๋ฆฌ์™€ ๋ณ‘๋ ฌ ํ”„๋กœ๊ทธ๋ž˜๋ฐ์„ ํ™œ์šฉํ•œ ๋ฐฉ๋ฒ•์œผ๋กœ ๊ฐœ์„ ์„ ์‹œ๋„ํ–ˆ์Šต๋‹ˆ๋‹ค. ์ด ๊ธ€์—์„œ๋Š” ๊ธฐ์กด ๋™๊ธฐ์ ์ธ ์ ‘๊ทผ ๋ฐฉ์‹์—์„œ ๊ฒช์—ˆ๋˜ ๋ฌธ์ œ์™€ ๊ทธ๋กœ ์ธํ•ด ๋ฐœ์ƒํ•œ ์ง€์—ฐ ๋ฌธ์ œ๋ฅผ ์–ด๋–ป๊ฒŒ ํ•ด๊ฒฐํ–ˆ๋Š”์ง€, ๊ทธ๋ฆฌ๊ณ  ์ƒˆ๋กœ์šด ๋ฐฉ๋ฒ•์œผ๋กœ ๊ฐœ์„ ๋œ ํšจ์œจ์„ฑ์— ๋Œ€ํ•ด ๊ณต์œ ํ•˜๋ ค ํ•ฉ๋‹ˆ๋‹ค. 

 

๋ฌธ์ œ ์ƒํ™ฉ

@Component
@RequiredArgsConstructor
public class Yes24Crawler implements Crawler {

	private final Fetcher fetcher;
	private final Parser parser;
	
	@Override
	public Map<Long, CrawledBook> crawlBestSellers() {
		Map<Long, String> bestSellerIds = parser.
			parseBestSellerIds(fetcher.fetchBestSellerIds());
		
		return bestSellerIds.entrySet() 
			.stream()
			.collect(Collectors.toMap(
				Map.Entry::getKey,
				entry -> parser.parseBook(fetcher.fetchBook(entry.getValue()))
			));
	}
	
}

ํ•ด๋‹น ์ฝ”๋“œ๋Š” parser.parseBestSellerIds(...) ๋ฉ”์„œ๋“œ๋ฅผ ํ†ตํ•ด ํฌ๋กค๋ง ๋œ 100๊ถŒ์˜ ๋„์„œ ์•„์ด๋””๋ฅผ ํ†ตํ•ด ์ƒ์„ธ ๋„์„œ ์ •๋ณด๋ฅผ ํฌ๋กค๋ง ํ•˜์—ฌ ์ˆœ์œ„ ์ •๋ณด์™€ ํ•จ๊ป˜ Map์œผ๋กœ ๋งคํ•‘ํ•˜์—ฌ ๋ฆฌํ„ดํ•ด์ฃผ๋Š” ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค. 

 

์•„๋ž˜ ์‚ฌ์ง„์€ ์œ„ ์ฝ”๋“œ์˜ ์ˆ˜ํ–‰ ์‹œ๊ฐ„์ž…๋‹ˆ๋‹ค. 

38sec 828ms์˜ ์ˆ˜ํ–‰์‹œ๊ฐ„์„ ๊ฐ€์ง„๋‹ค.

 

์™œ ๋ฐœ์ƒํ•˜๋‚˜์š”?

ํ˜„์žฌ ์ฝ”๋“œ์—์„œ์˜ ๋ฌธ์ œ์ ์€ ๋ชจ๋“  ๋„์„œ์— ๋Œ€ํ•œ HTTP ์š”์ฒญ์ด ์ˆœ์ฐจ์ ์œผ๋กœ ์ฒ˜๋ฆฌ๋˜๊ธฐ ๋•Œ๋ฌธ์— ํฌ๋กค๋ง ์‹œ๊ฐ„์ด ๋น„ํšจ์œจ์ ์œผ๋กœ ์†Œ์š”๋œ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค. 

 

๋™๊ธฐ์ ์ธ ์ˆ˜ํ–‰๋ฐฉ์‹

์ฆ‰, ์ฒซ ๋ฒˆ์งธ ๋„์„œ์˜ ์ •๋ณด๋ฅผ ๊ฐ€์ ธ์˜จ ํ›„์—์•ผ ๋‘ ๋ฒˆ์งธ ๋„์„œ์˜ ์ •๋ณด๋ฅผ ๊ฐ€์ ธ์˜ค๋Š” ๋ฐฉ์‹์œผ๋กœ ์ฒ˜๋ฆฌ๋˜๊ณ  ์žˆ์–ด, ์ „์ฒด์ ์ธ ์ฒ˜๋ฆฌ ์‹œ๊ฐ„์ด ํฌ๊ฒŒ ์ฆ๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ์ด๋Ÿฌํ•œ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด, ๋ณ‘๋ ฌ ์ฒ˜๋ฆฌ๋ฅผ ๋„์ž…ํ•˜์—ฌ HTTP ์š”์ฒญ์„ ๋™์‹œ์— ์ˆ˜ํ–‰ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๊ณ ๋ คํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๊ฐ ๋„์„œ์˜ ์ •๋ณด๋ฅผ ๋…๋ฆฝ์ ์œผ๋กœ ๊ฐ€์ ธ์˜ค๋ฉด์„œ ์ฒ˜๋ฆฌ ์‹œ๊ฐ„์„ ์ค„์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. 

 

ํ•ด๊ฒฐ ๋ฐฉ๋ฒ•

CompletableFuture

CompletableFuture๋Š” ๊ธฐ์กด์˜ Future ์ธํ„ฐํŽ˜์ด์Šค๊ฐ€ ์ œ๊ณตํ•˜์ง€ ๋ชปํ–ˆ๋˜ ๋น„๋™๊ธฐ ๊ฒฐ๊ณผ ๊ฐ’์˜ ์กฐํ•ฉ๊ณผ ์˜ˆ์™ธ ์ฒ˜๋ฆฌ๋ฅผ ํ›จ์”ฌ ๋” ํšจ๊ณผ์ ์œผ๋กœ ์ˆ˜ํ–‰ํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐ๋Šฅ์„ ๊ฐ–์ถ˜ ์ธํ„ฐํŽ˜์ด์Šค์ž…๋‹ˆ๋‹ค. ์ด ์ธํ„ฐํŽ˜์ด์Šค๋ฅผ ํ™œ์šฉํ•˜๋ฉด ๋™๊ธฐ ์ฝ”๋“œ๋ฅผ ๋น„๋™๊ธฐ ์ฝ”๋“œ๋กœ ๋ณ€ํ™˜ํ•˜์—ฌ ์„ฑ๋Šฅ์„ ๊ฐœ์„ ํ•˜๊ณ , ๋” ์œ ์—ฐํ•œ ๋น„๋™๊ธฐ ์ฒ˜๋ฆฌ ๋กœ์ง์„ ๊ตฌํ˜„ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. 

 

์ด์ œ, CompletableFuture๋ฅผ ํ™œ์šฉํ•œ ์ฝ”๋“œ ์ˆ˜์ •์„ ํ†ตํ•ด ๊ทธ ๊ณผ์ •์„ ์‚ดํŽด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค. 

@Component
@RequiredArgsConstructor
public class Yes24Crawler implements Crawler {
	
	private final Fetcher fetcher;
	private final Parser parser;
	
	@Override
	public Map<Long, CrawledBook> crawlBestSellers() {
		Map<Long, String> bestSellerIds = parser.parseBestSellerIds(fetcher.fetchBestSellerIds());
		Map<Long, CrawledBook> crawledBookMap = new ConcurrentHashMap<>();
		
		List<CompletableFuture<Void>> futures = bestSellerIds.entrySet().stream()
			.map(entry -> CompletableFuture.supplyAsync(() -> parser.parseBook(fetcher.fetchBook(entry.getValue())))
					.thenAccept(crawledBook -> crawledBookMap.put(entry.getKey(), crawledBook))
			.toList();
		
		CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();
		
		return crawledBookMap;
	}
	
}

๋ฒ ์ŠคํŠธ ์…€๋Ÿฌ ID ํŒŒ์‹ฑ

Map<Long, String> bestSellerIds = parser.parseBestSellerIds(fetcher.fetchBestSellerIds());

ํ•ด๋‹น ๋ฉ”์„œ๋“œ๋ฅผ ํ†ตํ•ด ๋ฒ ์ŠคํŠธ ์…€๋Ÿฌ ID๋ฅผ ํŒŒ์‹ฑํ•˜์—ฌ Map<Long, String>์— ํ• ๋‹นํ•ฉ๋‹ˆ๋‹ค. ์ด๋Š” ๋™๊ธฐ์ ์œผ๋กœ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. 

 

๋น„๋™๊ธฐ ๋ฐ์ดํ„ฐ ์ฒ˜๋ฆฌ

bestSellerIds.entrySet().stream()
	.map(entry -> CompletableFuture.supplyAsync(() -> parser.parseBook(fetcher.fetchBook(entry.getValue()))))
		.thenAccept(crawledBook -> crawledBookMap.put(entry.getKey(), crawledBook))
	.toList();

bestSellerIds์˜ ๊ฐ ํ•ญ๋ชฉ์— ๋Œ€ํ•ด CompletableFuture.supplyAsync(...)๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๋น„๋™๊ธฐ ์ž‘์—…์„ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค. ์ด ์ž‘์—…์€ fetcher.fetchBook(...)์„ ํ†ตํ•ด ํฌ๋กค๋งํ•œ ์ƒ์„ธ ๋„์„œ ์ •๋ณด๋ฅผ ๊ฐ€์ ธ์˜ค๊ณ , parser.parseBook(...)์„ ํ†ตํ•ด ์ฑ… ์ •๋ณด๋ฅผ ํŒŒ์‹ฑํ•ฉ๋‹ˆ๋‹ค. 

 

๊ทธ ํ›„ ๊ฐ ๋น„๋™๊ธฐ ์ž‘์—…์ด ์™„๋ฃŒ๋˜๋ฉด thenAccept(...)๋ฅผ ํ†ตํ•ด crawledBookMap์— ๊ฒฐ๊ณผ๋ฅผ ์ €์žฅํ•ฉ๋‹ˆ๋‹ค. 

 

๋น„๋™๊ธฐ ์ž‘์—… ๋Œ€๊ธฐ

CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();

CompletableFuture.allOf(...)๋Š” ๋ชจ๋“  ๋น„๋™๊ธฐ ์ž‘์—…์ด ์™„๋ฃŒ๋  ๋•Œ ๊นŒ์ง€ ๊ธฐ๋‹ค๋ฆฌ๋Š” ์ƒˆ๋กœ์šด CompletableFuture๋ฅผ ๋ฐ˜ํ™˜ํ•ฉ๋‹ˆ๋‹ค. ํ•ด๋‹น ๋ฉ”์„œ๋“œ๋ฅผ ํ†ตํ•ด CompletableFuture๊ฐ€ ๋ธ”๋กœํ‚น ๋˜๋Š” ๊ฒƒ ๊ฐ™์ง€๋งŒ ๊ทธ CompletableFuture๊ฐ€ ์‹ค์ œ๋กœ ์™„๋ฃŒ๋  ๋•Œ๊นŒ์ง€ ๋ธ”๋กœํ‚นํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. 

 

๋”ฐ๋ผ์„œ join()์„ ํ˜ธ์ถœํ•˜์—ฌ CompletableFuture.allOf()๊ฐ€ ๋ฐ˜ํ™˜ํ•œ CompletableFuture์˜ ์™„๋ฃŒ๋ฅผ ๊ธฐ๋‹ค๋ ค์•ผ ๋ชจ๋“  ๋น„๋™๊ธฐ ์ž‘์—…์ด ์™„๋ฃŒ๋œ ์ƒํƒœ๋ฅผ ๋ณด์žฅํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ ‡์ง€ ์•Š์œผ๋ฉด ๋น„๋™๊ธฐ ์ž‘์—…์ด ์™„๋ฃŒ๋˜๊ธฐ ์ „์— ๋ฉ”์ธ ์Šค๋ ˆ๋“œ๊ฐ€ ์ข…๋ฃŒ๋˜์–ด ๊ฒฐ๊ณผ๋ฅผ ์ œ๋Œ€๋กœ ์–ป์ง€ ๋ชปํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. 

 

๊ฒฐ๊ณผ

๋น„๋™๊ธฐ๋ฅผ ์ ์šฉํ•œ ๋ถ€๋ถ„

์œ„์™€ ๊ฐ™์ด ๊ฐœ์„ ๋œ ์ƒํ™ฉ์—์„œ ์ˆ˜ํ–‰ ์‹œ๊ฐ„์€ ์–ผ๋งˆ๋‚˜ ๋‹จ์ถ• ๋˜์—ˆ๋Š”์ง€ ํ™•์ธํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

 

@Component
@RequiredArgsConstructor
public class Yes24Crawler implements Crawler {

	private final Fetcher fetcher;
	private final Parser parser;
	
	@Override
	public Map<Long, CrawledBook> crawlBestSellers() {
		Map<Long, String> bestSellerIds = parser.
			parseBestSellerIds(fetcher.fetchBestSellerIds());
		
		return bestSellerIds.entrySet() 
			.stream()
			.collect(Collectors.toMap(
				Map.Entry::getKey,
				entry -> parser.parseBook(fetcher.fetchBook(entry.getValue()))
			));
	}
	
}

๋™๊ธฐ ๋ฐฉ์‹์œผ๋กœ ๋™์ž‘ํ•˜๋Š” ๊ธฐ์กด ๋ฐฉ์‹

 

@Component
@RequiredArgsConstructor
public class Yes24Crawler implements Crawler {
	
	private final Fetcher fetcher;
	private final Parser parser;
	
	@Override
	public Map<Long, CrawledBook> crawlBestSellers() {
		Map<Long, String> bestSellerIds = parser.parseBestSellerIds(fetcher.fetchBestSellerIds());
		Map<Long, CrawledBook> crawledBookMap = new ConcurrentHashMap<>();
		
		List<CompletableFuture<Void>> futures = bestSellerIds.entrySet().stream()
			.map(entry -> CompletableFuture.supplyAsync(() -> parser.parseBook(fetcher.fetchBook(entry.getValue())))
					.thenAccept(crawledBook -> crawledBookMap.put(entry.getKey(), crawledBook))
			.toList();
		
		CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();
		
		return crawledBookMap;
	}
	
}

CompletableFuture๋ฅผ ํ™œ์šฉํ•œ ๋น„๋™๊ธฐ ๋ฐฉ์‹

์ˆ˜ํ–‰ ๊ฒฐ๊ณผ๋ฅผ ํ†ตํ•ด์„œ ๋น„๋™๊ธฐ ํ”„๋กœ๊ทธ๋ž˜๋ฐ์—์„œ 1,006%์˜ ์„ฑ๋Šฅ ๊ฐœ์„ ์ด ๋˜์—ˆ์Œ์„ ๋ณผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. 

'Project' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€

๋Œ€์šฉ๋Ÿ‰ ๋ฐ์ดํ„ฐ๋กœ ์ธํ•œ API ์„ฑ๋Šฅ ๋ฌธ์ œ ํ•ด๊ฒฐ: ์ปค๋ฒ„๋ง ์ธ๋ฑ์Šค ๋ฐ ์ธ๋ฑ์Šค ํŠœ๋‹์„ ํ†ตํ•œ ์ตœ์ ํ™”  (1) 2025.05.29
GitHub Actions + Docker + NGINX๋ฅผ ํ™œ์šฉํ•œ Blue/Green ๋ฌด์ค‘๋‹จ ๋ฐฐํฌ ํ™˜๊ฒฝ ๊ตฌ์ถ•  (1) 2025.05.04
์ฑ„ํŒ… ๋ฉ”์„ธ์ง€ ์ฝ์Œ ์ฒ˜๋ฆฌ ๊ธฐ๋Šฅ ๊ตฌ์กฐ ๊ฐœ์„ ๊ธฐ  (0) 2024.12.18
Embedded Mongo/Redis ์ ์šฉํ•˜๊ธฐ  (0) 2024.11.26
๋ณ„์  ์ค‘๋ณต ์ƒ์„ฑ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐ  (0) 2024.10.27
'Project' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€
  • GitHub Actions + Docker + NGINX๋ฅผ ํ™œ์šฉํ•œ Blue/Green ๋ฌด์ค‘๋‹จ ๋ฐฐํฌ ํ™˜๊ฒฝ ๊ตฌ์ถ•
  • ์ฑ„ํŒ… ๋ฉ”์„ธ์ง€ ์ฝ์Œ ์ฒ˜๋ฆฌ ๊ธฐ๋Šฅ ๊ตฌ์กฐ ๊ฐœ์„ ๊ธฐ
  • Embedded Mongo/Redis ์ ์šฉํ•˜๊ธฐ
  • ๋ณ„์  ์ค‘๋ณต ์ƒ์„ฑ ๋™์‹œ์„ฑ ์ด์Šˆ ํ•ด๊ฒฐ
jwooo๐ŸŒฑ
jwooo๐ŸŒฑ
jwooo's log ์ž…๋‹ˆ๋‹ค.
  • jwooo๐ŸŒฑ
    jwooo's log
    jwooo๐ŸŒฑ
  • ์ „์ฒด
    ์˜ค๋Š˜
    ์–ด์ œ
    • ๋ถ„๋ฅ˜ ์ „์ฒด๋ณด๊ธฐ (12)
      • Java (4)
      • Project (6)
      • Computer Science (2)
        • Network (1)
        • Security (1)
  • ๋ธ”๋กœ๊ทธ ๋ฉ”๋‰ด

    • ํ™ˆ
    • ๋ฐฉ๋ช…๋ก
  • ๋งํฌ

  • ๊ณต์ง€์‚ฌํ•ญ

  • ์ธ๊ธฐ ๊ธ€

  • ์ตœ๊ทผ ๋Œ“๊ธ€

  • ์ตœ๊ทผ ๊ธ€

  • hELLOยท Designed By์ •์ƒ์šฐ.v4.10.3
jwooo๐ŸŒฑ
CompletableFuture๋ฅผ ํ™œ์šฉํ•œ ์„ฑ๋Šฅ ๊ฐœ์„ ๊ธฐ
์ƒ๋‹จ์œผ๋กœ

ํ‹ฐ์Šคํ† ๋ฆฌํˆด๋ฐ”