Wiki source code of WFM - Workflow Manager

Last modified by Kevin Austin on 2018/11/16 02:42

Show last authors
1 This project holds the Storm topologies that are used to implement the workflow management aspects of Kilda.
2
3 = Deployment =
4
5 To start topology in local mode:
6
7 (% class="box" %)
8 (((
9 (% style="font-family:Courier New,Courier,monospace" %)java -cp JAR.FILE CLASS ~-~-local arguments...
10 )))
11
12 To submit topology into storm:
13
14 (% class="box" %)
15 (((
16 (% style="font-family:Courier New,Courier,monospace" %)storm jar JAR.FILE class arguments...
17 )))
18
19 == Topology CLI arguments ==
20
21 In topology submit command you can pass arguments that will be passed into "main" call.
22
23 Supported arguments list:
24
25 * (% style="font-family:Courier New,Courier,monospace" %)~-~-name=NAME(%%) - override topology name. If don't passed topology name constructed from topology class name.
26 * (% style="font-family:Courier New,Courier,monospace" %)~-~-local(%%) - inform topology that is must run into "local" mode. I.e. submit topology into org.apache.storm.LocalCluster.
27 * (% style="font-family:Courier New,Courier,monospace" %)~-~-local-execution-time=TIME(%%) - used only in combination with (% style="font-family:Courier New,Courier,monospace" %)~-~-local(%%). Define how long (in seconds) topology will be executed. (% style="font-family:Courier New,Courier,monospace" %)TIME(%%) argument parsed as float number.
28 * (% style="font-family:Courier New,Courier,monospace" %)CONFIG(%%) - path to properties file. This properties will override compiled in properties. You can pass more than one file to pass override on override on override... and so on.
29
30 == Configuration ==
31
32 All topology options are defined as properties. There is compiled in JAR set of properties that provide a reasonable default values. You can pass more properties files(via CLI) to override this defaults.
33
34 Properties have different scope that depends from used prefix. Lets show it on option opentsdb.hosts.
35
36 * (% style="font-family:Courier New,Courier,monospace" %)opentsdb.hosts(%%) - this is global scope that will be used by all topologies
37 * (% style="font-family:Courier New,Courier,monospace" %)$name.opentsdb.hosts(%%) - this is name scope. (% style="font-family:Courier New,Courier,monospace" %)$name(%%) is passed from CLI (% style="font-family:Courier New,Courier,monospace" %)~-~-name(%%) argument. Or constructed from topology class name if name is missing.
38 * defaults.statstopology.opentsdb.hosts - this topology scope. In this scope, option (% style="font-family:Courier New,Courier,monospace" %)opentsdb.hosts(%%) will be used only by StatsTopology. Topology name statstopology constructed from topology class name.
39
40 Property lookup done in following order: (% style="font-family:Courier New,Courier,monospace" %)name scope, topology scope. global scope(%%). First found is used. This approach allow to pass options bounded to specific name, to specific topology and unbounded/globally.
41
42 = Developers =
43
44 == Debugging Tips ==
45
46 A lot of message passing is done through kafka topics.
47
48 === **WFM Debugging** ===
49
50 **Deploying the Kafka Splitter**
51
52 * assuming you've built the all-in-one jar, from within (% style="font-family:Courier New,Courier,monospace" %)services/wfm(%%):
53 * option 1:
54 * (% class="box" %)
55 (((
56 (% style="font-family:Courier New,Courier,monospace" %)mvn assembly:assembly -DskipTests
57 )))
58 * option 2:
59 * (% class="box" %)
60 (((
61 (% style="font-family:Courier New,Courier,monospace" %)mvn assembly:assembly
62 )))
63 * You can deploy the topology (with kilda running):
64 * (% class="box" %)
65 (((
66 (% style="font-family:Courier New,Courier,monospace" %)``` storm jar target/WorkflowManager-1.0-SNAPSHOT-jar-with-dependencies.jar \ org.openkilda.wfm.topology.event.OFEventSplitterTopology \ –name splitter-1 ```
67 )))
68
69 === Kafka Debugging ===
70
71 **Viewing Topics**
72
73 One way to look at what is going on in a topic:
74
75 Template:
76
77 * Example:
78 * (% class="box" %)
79 (((
80 (% style="font-family:Courier New,Courier,monospace" %)```kafka-console-consumer ~-~-bootstrap-server localhost:9092 ~-~-topic kilda.speaker ~-~-from-beginnin
81 )))
82
83 **Producing Messages on Topics**
84
85 Example:
86
87 * (% class="box" %)
88 (((
89 (% style="font-family:Courier New,Courier,monospace" %)kafka-console-producer ~-~-broker-list localhost:9092 ~-~-topic kilda.speaker
90 )))
91
92 === Storm Debugging ===
93
94 **Configuring Developer Environment**
95
96 * On the mac, you can use brew to install storm. That'll give you the (% style="font-family:Courier New,Courier,monospace" %)storm(%%) CLI.
97 * Ensure you configure /.storm/storm.yaml so that it points to your storm cluster.
98 ** Normally, this is the kilda cluster.
99 ** The storm.yaml file should look like this at a minimum:
100 ** (% class="box" %)
101 (((
102 (% style="font-family:Courier New,Courier,monospace" %)``` nimbus.seeds: ["127.0.0.1"] ```
103 )))
104 * With the CLI installed, you should be able to run the following kinds of commands:
105 * (% class="box" %)
106 (((
107 (% style="font-family:Courier New,Courier,monospace" %)``` storm list storm jar <jar.file> <class> [arguments] # ie deploy a topology stork kill <topology name>=""> ```
108 )))
109
110 **Viewing Logs**
111
112 Whereas you should be able to look at logs through the storm UI (ie localhost:8888), you can also look at the log files directly on the storm cluster:
113
114 * connect to the supervisor:
115 * (% class="box" %)
116 (((
117 (% style="font-family:Courier New,Courier,monospace" %)docker-compose exec storm-supervisor /bin/bash
118 )))
119 * cd to the base of the workers
120 * (% class="box" %)
121 (((
122 (% style="font-family:Courier New,Courier,monospace" %)cd /opt/storm/logs/workers-artifacts
123 )))
124 ** **Note: NB: (% style="font-family:Courier New,Courier,monospace" %)workers-artifacts(%%) will exist if you've deployed a topology; otherwise maybe.**
125
126 == Testing tips ==
127
128 **Unit tests**
129
130 Running JUnit target with coverage from the IDE builds coverage and should integrate data with the sources.
131
132 The command (% style="font-family:Courier New,Courier,monospace" %)mvn clean package(%%) builds unit tests and code coverage reports for using by external apps.
133
134 (% class="box" %)
135 (((
136 (% style="font-family:Courier New,Courier,monospace" %)mvn clean package
137 )))
138
139 Generated reports are stored in:
140
141 * (% style="font-family:Courier New,Courier,monospace" %)target/jacoco (%%)- jacoco reports
142 * (% style="font-family:Courier New,Courier,monospace" %)target/junit (%%)- junit reports
143 * (% style="font-family:Courier New,Courier,monospace" %)target/coverage-reports(%%) - coverage data files
144
145 Jacoco generates a site-package in (% style="font-family:Courier New,Courier,monospace" %)target/jacoco(%%) directory. It could be opened in any browser (% style="font-family:Courier New,Courier,monospace" %)(target/jacoco/index.html(%%)).
146
147 Junit reports could be used by external services (CI/CD for example) to represent test results.
148
149 Generated coverage data from the (% style="font-family:Courier New,Courier,monospace" %)target/coverage-reports(%%) directory could also be opened in IDE to show the coverage.
©2018 OpenKilda