1. StormDistributed and fault-toler

1. StormDistributed and fault-tolerant realtime computation system Chandler@PyHug previa [at] gmail.com
2. Outline• Background• Why Strom• Component• Topology• Storm & DRPC• Multilang Protocol• Experience
3. Background
4. Background• Creates by Nathan Marz @ BackType/Twitter – Analyze twits, links, users on Twitter• Opensourced at Sep 2011 – Eclipse Public License 1.0 – Storm 0.5.2 – 16k java and 7k Clojure Loc – Current stable release 0.8.2 • 0.9.0 major core improvement
5. Background• Active user group – https://groups.google.com/group/storm-user – https://github.com/nathanmarz/storm – Most watched java repo at GitHub (>4k watcher) – Used by over 30 companies • Twitter, Groupon, Alibaba, GumGum, ..
6. Why Storm ?
7. Before Storm
8. Problems• Scale is painful• Poor fault-tolerance – Hadoop is stateful• Coding is tedious• Batch processing – Long latency – no realtime
9. Storm• Scalable and robust – No persistent layer• Guarantees no data loss• Fault-tolerant• Programming language agnostic• Use case – Stream processing – Distributed RPC – Continues computation
10. Components
11. Base on• Apache Zookeeper – Distributed system, used to store metadata• ØMQ – Asynchronous message transport layer• Apache Thrift – Cross-language bridge, RPC• LMAX Disruptor – High performance queue shared by threads• Kryo – Serialization framework
12. System architecture
13. System architecture• Nimbus – Like JobtTacker in hadoop• Supervisor – Manage workers• Zookeeper – Store meta data• UI – Web-UI
14. Topology
15. Topology• Tuples – ordered list of elements – (“user”, “link”, “event”, “10/3/12 17:50“)• Streams – unbounded sequence of tuples
16. Spouts• Source of streams• Example • Read from logs, API calls, event data, queues, …
17. Spout• Interface ISpout – BaseRichSpout, ClojureSpout, DRPCSpout, FeederSpout, FixedTupleSpout, MasterBatchCoordinator, NoOpSpout, RichShellSpout, RichSpoutBatchTriggerer, ShellS pout, SpoutTracker, TestPlannerSpout, TestWordSpout, TransactionalSpoutCoordinator
18. Topology• Bolts – Processes input streams and produces new streams – Example • Stream Joins, DBs, APIs, Filters, Aggregation, …
19. Bolts• Interface Ibolt – BaseRichBolt, BasicBoltExecutor, BatchBoltExecutor, BoltTracker, ClojureBolt, Coordinate dBolt, JoinResult, KeyedFairBolt, NonRichBoltTracker, ReturnResults, BaseShellBolt, ShellBolt, TestAggregatesCounter, TestGlobalCount, TestPlannerBolt, TransactionalSpout BatchExecutor,TridentBoltExecutor, TupleCaptureBolt
20. Topology• Topology – A directed graph of Spouts and Bolts
21. Tasks• Instances of Spouts and Blots• Managed by Supervisor – http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/
22. Stream grouping• All grouping – Send to all tasks• Global grouping – Pick task with lowest id• Shuffle grouping – Pick a random task• Fields grouping – Consistent hashing on a subset of tuple fields
23. Storm fault-tolerance• Reliability API – Spout tuple creation • colloctor.emit(values, msgID); – Child tuple creation (Bolts) • colloctor.emit(parentTuples, values); – Tuple end of processing • collector.ack(tuple); – Tuple failed to process • collector.fail(tuple);
24. Storm fault-tolerance• Disable reliability API – Globally • Config.TOPOLOGY_ACKER_EXECUTORS = 0 – On topology level • Collector.emit(values, msgID); – For a single tuple • Collector.emit(paranetTuples, values);
25. Storm & DRPC
26. Distributed RPC
27. Multilang Protocol
28. Multilang protocol• Using ShellSpout/ShellBolt• Process using stand in/out to communicate• Massage are encoded as JSON/ lines of plain text
29. Three steps• Initiate a handshake – Keep track with process id – Send a json object to standard input while start – Contains • Storm configuration, topology, context, PID directory
30. Three steps• Start looping – storm_sync would expect torm_ack• Read or write tuples – Follow defined structure – Implement read_msg(), storm_emit() ,…
31. Experience
32. Experience• Not hard to setup, but – Beware of certain version of Zookeeper – Wait a while after topology deployed• Fast, – Better use fabric• Stable – But beware of memory leak

ผลลัพธ์ (ไทย) 1:

1. StormDistributed และเรียลไทม์ทนต่อเกาะต่ำคำนวณ Chandler@PyHug เพื่อระบบ [ที่] gmail.com2. ทำไมพายุ Topology• Component• Strom• และ DRPC• Multilang Protocol• ประสบการณ์ Background• Outline•3. พื้นหลัง4. Background• สร้าง โดย Marz นาธาน @ BackType/Twitter –วิเคราะห์ twits ลิงค์ ผู้ Opensourced Twitter• ที่ Sep 2011 –คราสสาธารณะใบอนุญาต 1.0 – พายุโหด 0.5.2 – 16 k java และ 7k Clojure ล็อค – ปล่อยมั่นคงปัจจุบัน 0.8.2 พัฒนาหลักสำคัญ• 0.9.05. Background• ใช้งานผู้ใช้กลุ่ม – https://groups.google.com/group/storm-user – https://github.com/nathanmarz/storm – ชวา repo ที่ GitHub เฝ้าดูมากที่สุด (> 4 k watcher) – ใช้ โดยบริษัทกว่า 30 • Twitter, Groupon อาลีบาบา GumGum, ...6. ทำไมพายุ7. ก่อนพายุ8. Problems• ขนาด painful• ไม่ดีบกพร่องยอมรับ – stateful• เป็นอย่างไร Hadoop Coding เป็น tedious• ชุดประมวลผล –แฝงยาว – ไม่เรียลไทม์9. Storm• ปรับสเกล และแข็งแกร่ง-layer• แบบไม่รับประกันไม่ข้อมูล loss• tolerant• ข้อบกพร่องการเขียนโปรแกรมภาษา agnostic• ใช้กรณี – กระแสข้อมูลประมวลผล –กระจาย RPC – Continues คำนวณ10. ประกอบ11. ฐาน on• Apache Zookeeper – Distributed ระบบ ใช้เก็บ metadata• ØMQ –ข้อแบบอะซิงโครนัสขนส่ง layer• เกี่ยวกับภัตตาคาร Apache – สะพานข้ามภาษา RPC• LMAX Disruptor-ประสิทธิภาพสูงคิวร่วม threads• Kryo – อนุกรมกรอบ12. System architecture13. System architecture• Nimbus – Like JobtTacker in hadoop• Supervisor – Manage workers• Zookeeper – Store meta data• UI – Web-UI14. Topology15. Topology• Tuples – ordered list of elements – (“user”, “link”, “event”, “10/3/12 17:50“)• Streams – unbounded sequence of tuples16. Spouts• Source of streams• Example • Read from logs, API calls, event data, queues, …17. Spout• Interface ISpout – BaseRichSpout, ClojureSpout, DRPCSpout, FeederSpout, FixedTupleSpout, MasterBatchCoordinator, NoOpSpout, RichShellSpout, RichSpoutBatchTriggerer, ShellS pout, SpoutTracker, TestPlannerSpout, TestWordSpout, TransactionalSpoutCoordinator18. Topology• Bolts – Processes input streams and produces new streams – Example • Stream Joins, DBs, APIs, Filters, Aggregation, …19. Bolts• Interface Ibolt – BaseRichBolt, BasicBoltExecutor, BatchBoltExecutor, BoltTracker, ClojureBolt, Coordinate dBolt, JoinResult, KeyedFairBolt, NonRichBoltTracker, ReturnResults, BaseShellBolt, ShellBolt, TestAggregatesCounter, TestGlobalCount, TestPlannerBolt, TransactionalSpout BatchExecutor,TridentBoltExecutor, TupleCaptureBolt20. Topology• Topology – A directed graph of Spouts and Bolts21. Tasks• Instances of Spouts and Blots• Managed by Supervisor – http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/22. Stream grouping• All grouping – Send to all tasks• Global grouping – Pick task with lowest id• Shuffle grouping – Pick a random task• Fields grouping – Consistent hashing on a subset of tuple fields23. Storm fault-tolerance• Reliability API – Spout tuple creation • colloctor.emit(values, msgID); – Child tuple creation (Bolts) • colloctor.emit(parentTuples, values); – Tuple end of processing • collector.ack(tuple); – Tuple failed to process • collector.fail(tuple);24. Storm fault-tolerance• Disable reliability API – Globally • Config.TOPOLOGY_ACKER_EXECUTORS = 0 – On topology level • Collector.emit(values, msgID); – For a single tuple • Collector.emit(paranetTuples, values);25. Storm & DRPC26. Distributed RPC27. Multilang Protocol28. Multilang protocol• Using ShellSpout/ShellBolt• Process using stand in/out to communicate• Massage are encoded as JSON/ lines of plain text29. Three steps• Initiate a handshake – Keep track with process id – Send a json object to standard input while start – Contains • Storm configuration, topology, context, PID directory30. Three steps• Start looping – storm_sync would expect torm_ack• Read or write tuples – Follow defined structure – Implement read_msg(), storm_emit() ,…31. Experience32. Experience• Not hard to setup, but – Beware of certain version of Zookeeper – Wait a while after topology deployed• Fast, – Better use fabric• Stable – But beware of memory leak