首页 探索 帮助
注册 登录
J1X
/
huginn
关注 2
点赞 0
派生 0

浏览代码
Merge pull request #98 from cantino/human_task_agent_makes_polls

HumanTaskAgent can now validate answers with a poll

Andrew Cantino 11 年之前
父节点
8fde7492ca b32cef09c9
当前提交
5461ec3d2f
共有 2 个文件被更改，包括 369 次插入 和 62 次删除
显示文件统计
  1. 198 44
      app/models/agents/human_task_agent.rb
  2. 171 18
      spec/models/agents/human_task_agent_spec.rb

+ 198 - 44
app/models/agents/human_task_agent.rb
查看文件 

@@ -9,9 +9,13 @@ module Agents
9 9
10 10 
       HITs can be created in response to events, or on a schedule.  Set `trigger_on` to either `schedule` or `event`.
11 11
12 
+      # Schedule
13 
+
12 14 
       The schedule of this Agent is how often it should check for completed HITs, __NOT__ how often to submit one.  To configure how often a new HIT
13 15 
       should be submitted when in `schedule` mode, set `submission_period` to a number of hours.
14 16
17 
+      # Example
18 
+
15 19 
       If created with an event, all HIT fields can contain interpolated values via [JSONPaths](http://goessner.net/articles/JsonPath/) placed between < and > characters.
16 20 
       For example, if the incoming event was a Twitter event, you could make a HITT to rate its sentiment like this:
17 21 
 
@@ -58,8 +62,52 @@ module Agents
58 62 
       which contain `key` and `text`.  For _free\\_text_, the special configuration options are all optional, and are
59 63 
       `default`, `min_length`, and `max_length`.
60 64
61 
-      If all of the `questions` are of `type` _selection_, you can set `take_majority` to _true_ at the top level to
62 
-      automatically select the majority vote for each question across all `assignments`.  If all selections are numeric, an `average_answer` will also be generated.
65 
+      # Combining answers
66 
+
67 
+      There are a couple of ways to combine HITs that have multiple `assignments`, all of which involve setting `combination_mode` at the top level.
68 
+
69 
+      ## Taking the majority
70 
+
71 
+      Option 1: if all of your `questions` are of `type` _selection_, you can set `combination_mode` to `take_majority`.
72 
+      This will cause the Agent to automatically select the majority vote for each question across all `assignments` and return it as `majority_answer`.
73 
+      If all selections are numeric, an `average_answer` will also be generated.
74 
+
75 
+      Option 2: you can have the Agent ask additional human workers to rank the `assignments` and return the most highly ranked answer.
76 
+      To do this, set `combination_mode` to `poll` and provide a `poll_options` object.  Here is an example:
77 
+
78 
+          {
79 
+            "trigger_on": "schedule",
80 
+            "submission_period": 12,
81 
+            "combination_mode": "poll",
82 
+            "poll_options": {
83 
+              "title": "Take a poll about some jokes",
84 
+              "instructions": "Please rank these jokes from most funny (5) to least funny (1)",
85 
+              "assignments": 3,
86 
+              "row_template": "<$.joke>"
87 
+            },
88 
+            "hit": {
89 
+              "assignments": 5,
90 
+              "title": "Tell a joke",
91 
+              "description": "Please tell me a joke",
92 
+              "reward": 0.05,
93 
+              "lifetime_in_seconds": "3600",
94 
+              "questions": [
95 
+                {
96 
+                  "type": "free_text",
97 
+                  "key": "joke",
98 
+                  "name": "Your joke",
99 
+                  "required": "true",
100 
+                  "question": "Joke",
101 
+                  "min_length": "2",
102 
+                  "max_length": "2000"
103 
+                }
104 
+              ]
105 
+            }
106 
+          }
107 
+
108 
+      Resulting events will have the original `answers`, as well as the `poll` results, and a field called `best_answer` that contains the best answer as determined by the poll.
109 
+
110 
+      # Other settings
63 111
64 112 
       `lifetime_in_seconds` is the number of seconds a HIT is left on Amazon before it's automatically closed.  The default is 1 day.
65 113 
 
@@ -70,6 +118,12 @@ module Agents
70 118 
       Events look like:
71 119
72 120 
           {
121 
+            "answers": [
122 
+              {
123 
+                "feedback": "Hello!",
124 
+                "sentiment": "happy"
125 
+              }
126 
+            ]
73 127 
           }
74 128 
     MD
75 129 
 
@@ -97,9 +151,13 @@ module Agents
97 151 
         errors.add(:base, "all questions of type 'selection' must have a selections array with selections that set 'key' and 'name'")
98 152 
       end
99 153
100 
-      if options[:take_majority] == "true" && options[:hit][:questions].any? { |question| question[:type] != "selection" }
154 
+      if take_majority? && options[:hit][:questions].any? { |question| question[:type] != "selection" }
101 155 
         errors.add(:base, "all questions must be of type 'selection' to use the 'take_majority' option")
102 156 
       end
157 
+
158 
+      if create_poll?
159 
+        errors.add(:base, "poll_options is required when combination_mode is set to 'poll' and must have the keys 'title', 'instructions', 'row_template', and 'assignments'") unless options[:poll_options].is_a?(Hash) && options[:poll_options][:title].present? &&  options[:poll_options][:instructions].present? && options[:poll_options][:row_template].present? && options[:poll_options][:assignments].to_i > 0
160 
+      end
103 161 
     end
104 162
105 163 
     def default_options
 
@@ -152,69 +210,152 @@ module Agents
152 210
153 211 
       if options[:trigger_on] == "schedule" && (memory[:last_schedule] || 0) <= Time.now.to_i - options[:submission_period].to_i * 60 * 60
154 212 
         memory[:last_schedule] = Time.now.to_i
155 
-        create_hit
213 
+        create_basic_hit
156 214 
       end
157 215 
     end
158 216
159 217 
     def receive(incoming_events)
160 218 
       if options[:trigger_on] == "event"
161 219 
         incoming_events.each do |event|
162 
-          create_hit event
220 
+          create_basic_hit event
163 221 
         end
164 222 
       end
165 223 
     end
166 224
167 225 
     protected
168 226
227 
+    def take_majority?
228 
+      options[:combination_mode] == "take_majority" || options[:take_majority] == "true"
229 
+    end
230 
+
231 
+    def create_poll?
232 
+      options[:combination_mode] == "poll"
233 
+    end
234 
+
235 
+    def event_for_hit(hit_id)
236 
+      if memory[:hits][hit_id.to_sym].is_a?(Hash)
237 
+        Event.find_by_id(memory[:hits][hit_id.to_sym][:event_id])
238 
+      else
239 
+        nil
240 
+      end
241 
+    end
242 
+
243 
+    def hit_type(hit_id)
244 
+      # Fix this: the Ruby process will slowly run out of RAM by symbolizing these unique keys.
245 
+      if memory[:hits][hit_id.to_sym].is_a?(Hash) && memory[:hits][hit_id.to_sym][:type]
246 
+        memory[:hits][hit_id.to_sym][:type].to_sym
247 
+      else
248 
+        :user
249 
+      end
250 
+    end
251 
+
169 252 
     def review_hits
170 253 
       reviewable_hit_ids = RTurk::GetReviewableHITs.create.hit_ids
171 254 
       my_reviewed_hit_ids = reviewable_hit_ids & (memory[:hits] || {}).keys.map(&:to_s)
172 255 
       if reviewable_hit_ids.length > 0
173 256 
         log "MTurk reports #{reviewable_hit_ids.length} HITs, of which I own [#{my_reviewed_hit_ids.to_sentence}]"
174 257 
       end
258 
+
175 259 
       my_reviewed_hit_ids.each do |hit_id|
176 260 
         hit = RTurk::Hit.new(hit_id)
177 261 
         assignments = hit.assignments
178 262
179 263 
         log "Looking at HIT #{hit_id}.  I found #{assignments.length} assignments#{" with the statuses: #{assignments.map(&:status).to_sentence}" if assignments.length > 0}"
180 264 
         if assignments.length == hit.max_assignments && assignments.all? { |assignment| assignment.status == "Submitted" }
181 
-          payload = { :answers => assignments.map(&:answers) }
182 
-
183 
-          if options[:take_majority] == "true"
184 
-            counts = {}
185 
-            options[:hit][:questions].each do |question|
186 
-              question_counts = question[:selections].inject({}) { |memo, selection| memo[selection[:key]] = 0; memo }
187 
-              assignments.each do |assignment|
188 
-                answers = ActiveSupport::HashWithIndifferentAccess.new(assignment.answers)
189 
-                answer = answers[question[:key]]
190 
-                question_counts[answer] += 1
265 
+          inbound_event = event_for_hit(hit_id)
266 
+
267 
+          if hit_type(hit_id) == :poll
268 
+            # handle completed polls
269 
+
270 
+            log "Handling a poll: #{hit_id}"
271 
+
272 
+            scores = {}
273 
+            assignments.each do |assignment|
274 
+              assignment.answers.each do |index, rating|
275 
+                scores[index] ||= 0
276 
+                scores[index] += rating.to_i
191 277 
               end
192 
-              counts[question[:key]] = question_counts
193 278 
             end
194 
-            payload[:counts] = counts
195 279
196 
-            majority_answer = counts.inject({}) do |memo, (key, question_counts)|
197 
-              memo[key] = question_counts.to_a.sort {|a, b| a.last <=> b.last }.last.first
198 
-              memo
199 
-            end
200 
-            payload[:majority_answer] = majority_answer
201 
-
202 
-            if all_questions_are_numeric?
203 
-              average_answer = counts.inject({}) do |memo, (key, question_counts)|
204 
-                sum = divisor = 0
205 
-                question_counts.to_a.each do |num, count|
206 
-                  sum += num.to_s.to_f * count
207 
-                  divisor += count
280 
+            top_answer = scores.to_a.sort {|b, a| a.last <=> b.last }.first.first
281 
+
282 
+            payload = {
283 
+              :answers => memory[:hits][hit_id.to_sym][:answers],
284 
+              :poll => assignments.map(&:answers),
285 
+              :best_answer => memory[:hits][hit_id.to_sym][:answers][top_answer.to_i - 1]
286 
+            }
287 
+
288 
+            event = create_event :payload => payload
289 
+            log "Event emitted with answer(s) for poll", :outbound_event => event, :inbound_event => inbound_event
290 
+          else
291 
+            # handle normal completed HITs
292 
+            payload = { :answers => assignments.map(&:answers) }
293 
+
294 
+            if take_majority?
295 
+              counts = {}
296 
+              options[:hit][:questions].each do |question|
297 
+                question_counts = question[:selections].inject({}) { |memo, selection| memo[selection[:key]] = 0; memo }
298 
+                assignments.each do |assignment|
299 
+                  answers = ActiveSupport::HashWithIndifferentAccess.new(assignment.answers)
300 
+                  answer = answers[question[:key]]
301 
+                  question_counts[answer] += 1
208 302 
                 end
209 
-                memo[key] = sum / divisor.to_f
303 
+                counts[question[:key]] = question_counts
304 
+              end
305 
+              payload[:counts] = counts
306 
+
307 
+              majority_answer = counts.inject({}) do |memo, (key, question_counts)|
308 
+                memo[key] = question_counts.to_a.sort {|a, b| a.last <=> b.last }.last.first
210 309 
                 memo
211 310 
               end
212 
-              payload[:average_answer] = average_answer
311 
+              payload[:majority_answer] = majority_answer
312 
+
313 
+              if all_questions_are_numeric?
314 
+                average_answer = counts.inject({}) do |memo, (key, question_counts)|
315 
+                  sum = divisor = 0
316 
+                  question_counts.to_a.each do |num, count|
317 
+                    sum += num.to_s.to_f * count
318 
+                    divisor += count
319 
+                  end
320 
+                  memo[key] = sum / divisor.to_f
321 
+                  memo
322 
+                end
323 
+                payload[:average_answer] = average_answer
324 
+              end
213 325 
             end
214 
-          end
215 326
216 
-          event = create_event :payload => payload
217 
-          log "Event emitted with answer(s)", :outbound_event => event, :inbound_event => Event.find_by_id(memory[:hits][hit_id.to_sym])
327 
+            if create_poll?
328 
+              questions = []
329 
+              selections = 5.times.map { |i| { :key => i+1, :text => i+1 } }.reverse
330 
+              assignments.length.times do |index|
331 
+                questions << {
332 
+                  :type => "selection",
333 
+                  :name => "Item #{index + 1}",
334 
+                  :key => index,
335 
+                  :required => "true",
336 
+                  :question => Utils.interpolate_jsonpaths(options[:poll_options][:row_template], assignments[index].answers),
337 
+                  :selections => selections
338 
+                }
339 
+              end
340 
+
341 
+              poll_hit = create_hit :title => options[:poll_options][:title],
342 
+                                    :description => options[:poll_options][:instructions],
343 
+                                    :questions => questions,
344 
+                                    :assignments => options[:poll_options][:assignments],
345 
+                                    :lifetime_in_seconds => options[:poll_options][:lifetime_in_seconds],
346 
+                                    :reward => options[:poll_options][:reward],
347 
+                                    :payload => inbound_event && inbound_event.payload,
348 
+                                    :metadata => { :type => :poll,
349 
+                                                   :original_hit => hit_id,
350 
+                                                   :answers => assignments.map(&:answers),
351 
+                                                   :event_id => inbound_event && inbound_event.id }
352 
+
353 
+              log "Poll HIT created with ID #{poll_hit.id} and URL #{poll_hit.url}.  Original HIT: #{hit_id}", :inbound_event => inbound_event
354 
+            else
355 
+              event = create_event :payload => payload
356 
+              log "Event emitted with answer(s)", :outbound_event => event, :inbound_event => inbound_event
357 
+            end
358 
+          end
218 359
219 360 
           assignments.each(&:approve!)
220 361 
           hit.dispose!
 
@@ -232,22 +373,35 @@ module Agents
232 373 
       end
233 374 
     end
234 375
235 
-    def create_hit(event = nil)
236 
-      payload = event ? event.payload : {}
237 
-      title = Utils.interpolate_jsonpaths(options[:hit][:title], payload).strip
238 
-      description = Utils.interpolate_jsonpaths(options[:hit][:description], payload).strip
239 
-      questions = Utils.recursively_interpolate_jsonpaths(options[:hit][:questions], payload)
376 
+    def create_basic_hit(event = nil)
377 
+      hit = create_hit :title => options[:hit][:title],
378 
+                       :description => options[:hit][:description],
379 
+                       :questions => options[:hit][:questions],
380 
+                       :assignments => options[:hit][:assignments],
381 
+                       :lifetime_in_seconds => options[:hit][:lifetime_in_seconds],
382 
+                       :reward => options[:hit][:reward],
383 
+                       :payload => event && event.payload,
384 
+                       :metadata => { :event_id => event && event.id }
385 
+
386 
+      log "HIT created with ID #{hit.id} and URL #{hit.url}", :inbound_event => event
387 
+    end
388 
+
389 
+    def create_hit(opts = {})
390 
+      payload = opts[:payload] || {}
391 
+      title = Utils.interpolate_jsonpaths(opts[:title], payload).strip
392 
+      description = Utils.interpolate_jsonpaths(opts[:description], payload).strip
393 
+      questions = Utils.recursively_interpolate_jsonpaths(opts[:questions], payload)
240 394 
       hit = RTurk::Hit.create(:title => title) do |hit|
241 
-        hit.max_assignments = (options[:hit][:assignments] || 1).to_i
395 
+        hit.max_assignments = (opts[:assignments] || 1).to_i
242 396 
         hit.description = description
243 
-        hit.lifetime = (options[:hit][:lifetime_in_seconds] || 24 * 60 * 60).to_i
397 
+        hit.lifetime = (opts[:lifetime_in_seconds] || 24 * 60 * 60).to_i
244 398 
         hit.question_form AgentQuestionForm.new(:title => title, :description => description, :questions => questions)
245 
-        hit.reward = (options[:hit][:reward] || 0.05).to_f
399 
+        hit.reward = (opts[:reward] || 0.05).to_f
246 400 
         #hit.qualifications.add :approval_rate, { :gt => 80 }
247 401 
       end
248 402 
       memory[:hits] ||= {}
249 
-      memory[:hits][hit.id] = event && event.id
250 
-      log "HIT created with ID #{hit.id} and URL #{hit.url}", :inbound_event => event
403 
+      memory[:hits][hit.id] = opts[:metadata] || {}
404 
+      hit
251 405 
     end
252 406
253 407 
     # RTurk Question Form

+ 171 - 18
spec/models/agents/human_task_agent_spec.rb
查看文件 

@@ -108,7 +108,43 @@ describe Agents::HumanTaskAgent do
108 108 
       @checker.should_not be_valid
109 109 
     end
110 110
111 
-    it "requires that all questions be of type 'selection' when `take_majority` is `true`" do
111 
+    it "requires that 'poll_options' be present and populated when 'combination_mode' is set to 'poll'" do
112 
+      @checker.options[:combination_mode] = "poll"
113 
+      @checker.should_not be_valid
114 
+      @checker.options[:poll_options] = {}
115 
+      @checker.should_not be_valid
116 
+      @checker.options[:poll_options] = { :title => "Take a poll about jokes",
117 
+                                          :instructions => "Rank these by how funny they are",
118 
+                                          :assignments => 3,
119 
+                                          :row_template => "<$.joke>" }
120 
+      @checker.should be_valid
121 
+      @checker.options[:poll_options] = { :instructions => "Rank these by how funny they are",
122 
+                                          :assignments => 3,
123 
+                                          :row_template => "<$.joke>" }
124 
+      @checker.should_not be_valid
125 
+      @checker.options[:poll_options] = { :title => "Take a poll about jokes",
126 
+                                          :assignments => 3,
127 
+                                          :row_template => "<$.joke>" }
128 
+      @checker.should_not be_valid
129 
+      @checker.options[:poll_options] = { :title => "Take a poll about jokes",
130 
+                                          :instructions => "Rank these by how funny they are",
131 
+                                          :row_template => "<$.joke>" }
132 
+      @checker.should_not be_valid
133 
+      @checker.options[:poll_options] = { :title => "Take a poll about jokes",
134 
+                                          :instructions => "Rank these by how funny they are",
135 
+                                          :assignments => 3}
136 
+      @checker.should_not be_valid
137 
+    end
138 
+
139 
+    it "requires that all questions be of type 'selection' when 'combination_mode' is 'take_majority'" do
140 
+      @checker.options[:combination_mode] = "take_majority"
141 
+      @checker.should_not be_valid
142 
+      @checker.options[:hit][:questions][1][:type] = "selection"
143 
+      @checker.options[:hit][:questions][1][:selections] = @checker.options[:hit][:questions][0][:selections]
144 
+      @checker.should be_valid
145 
+    end
146 
+
147 
+    it "accepts 'take_majority': 'true' for legacy support" do
112 148 
       @checker.options[:take_majority] = "true"
113 149 
       @checker.should_not be_valid
114 150 
       @checker.options[:hit][:questions][1][:type] = "selection"
 
@@ -126,7 +162,7 @@ describe Agents::HumanTaskAgent do
126 162
127 163 
     it "should check for reviewable HITs frequently" do
128 164 
       mock(@checker).review_hits.twice
129 
-      mock(@checker).create_hit.once
165 
+      mock(@checker).create_basic_hit.once
130 166 
       @checker.check
131 167 
       @checker.check
132 168 
     end
 
@@ -135,7 +171,7 @@ describe Agents::HumanTaskAgent do
135 171 
       now = Time.now
136 172 
       stub(Time).now { now }
137 173 
       mock(@checker).review_hits.times(3)
138 
-      mock(@checker).create_hit.twice
174 
+      mock(@checker).create_basic_hit.twice
139 175 
       @checker.check
140 176 
       now += 1 * 60 * 60
141 177 
       @checker.check
 
@@ -144,7 +180,7 @@ describe Agents::HumanTaskAgent do
144 180 
     end
145 181
146 182 
     it "should ignore events" do
147 
-      mock(@checker).create_hit(anything).times(0)
183 
+      mock(@checker).create_basic_hit(anything).times(0)
148 184 
       @checker.receive([events(:bob_website_agent_event)])
149 185 
     end
150 186 
   end
 
@@ -155,7 +191,7 @@ describe Agents::HumanTaskAgent do
155 191 
       now = Time.now
156 192 
       stub(Time).now { now }
157 193 
       mock(@checker).review_hits.times(3)
158 
-      mock(@checker).create_hit.times(0)
194 
+      mock(@checker).create_basic_hit.times(0)
159 195 
       @checker.check
160 196 
       now += 1 * 60 * 60
161 197 
       @checker.check
 
@@ -164,7 +200,7 @@ describe Agents::HumanTaskAgent do
164 200 
     end
165 201
166 202 
     it "should create HITs based on events" do
167 
-      mock(@checker).create_hit(events(:bob_website_agent_event)).times(1)
203 
+      mock(@checker).create_basic_hit(events(:bob_website_agent_event)).times(1)
168 204 
       @checker.receive([events(:bob_website_agent_event)])
169 205 
     end
170 206 
   end
 
@@ -181,7 +217,7 @@ describe Agents::HumanTaskAgent do
181 217 
       mock(hitInterface).question_form(instance_of Agents::HumanTaskAgent::AgentQuestionForm) { |agent_question_form_instance| question_form = agent_question_form_instance }
182 218 
       mock(RTurk::Hit).create(:title => "Hi Joe").yields(hitInterface) { hitInterface }
183 219
184 
-      @checker.send :create_hit, @event
220 
+      @checker.send :create_basic_hit, @event
185 221
186 222 
       hitInterface.max_assignments.should == @checker.options[:hit][:assignments]
187 223 
       hitInterface.reward.should == @checker.options[:hit][:reward]
 
@@ -192,7 +228,7 @@ describe Agents::HumanTaskAgent do
192 228 
       xml.should include("<Text>Make something for Joe</Text>")
193 229 
       xml.should include("<DisplayName>Joe Question 1</DisplayName>")
194 230
195 
-      @checker.memory[:hits][123].should == @event.id
231 
+      @checker.memory[:hits][123][:event_id].should == @event.id
196 232 
     end
197 233
198 234 
     it "works without an event too" do
 
@@ -201,7 +237,7 @@ describe Agents::HumanTaskAgent do
201 237 
       hitInterface.id = 123
202 238 
       mock(hitInterface).question_form(instance_of Agents::HumanTaskAgent::AgentQuestionForm)
203 239 
       mock(RTurk::Hit).create(:title => "Hi").yields(hitInterface) { hitInterface }
204 
-      @checker.send :create_hit
240 
+      @checker.send :create_basic_hit
205 241 
       hitInterface.max_assignments.should == @checker.options[:hit][:assignments]
206 242 
       hitInterface.reward.should == @checker.options[:hit][:reward]
207 243 
     end
 
@@ -259,8 +295,8 @@ describe Agents::HumanTaskAgent do
259 295
260 296 
       # It knows about two HITs from two different events.
261 297 
       @checker.memory[:hits] = {}
262 
-      @checker.memory[:hits][:"JH3132836336DHG"] = @event.id
263 
-      @checker.memory[:hits][:"JH39AA63836DHG"] = event2.id
298 
+      @checker.memory[:hits][:"JH3132836336DHG"] = { :event_id => @event.id }
299 
+      @checker.memory[:hits][:"JH39AA63836DHG"] = { :event_id => event2.id }
264 300
265 301 
       hit_ids = %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345]
266 302 
       mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { hit_ids } } # It sees 3 HITs.
 
@@ -273,7 +309,7 @@ describe Agents::HumanTaskAgent do
273 309 
     end
274 310
275 311 
     it "shouldn't do anything if an assignment isn't ready" do
276 
-      @checker.memory[:hits] = { :"JH3132836336DHG" => @event.id }
312 
+      @checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
277 313 
       mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
278 314 
       assignments = [
279 315 
         FakeAssignment.new(:status => "Accepted", :answers => {}),
 
@@ -288,11 +324,11 @@ describe Agents::HumanTaskAgent do
288 324 
       @checker.send :review_hits
289 325
290 326 
       assignments.all? {|a| a.approved == true }.should be_false
291 
-      @checker.memory[:hits].should == { :"JH3132836336DHG" => @event.id }
327 
+      @checker.memory[:hits].should == { :"JH3132836336DHG" => { :event_id => @event.id } }
292 328 
     end
293 329
294 330 
     it "shouldn't do anything if an assignment is missing" do
295 
-      @checker.memory[:hits] = { :"JH3132836336DHG" => @event.id }
331 
+      @checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
296 332 
       mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
297 333 
       assignments = [
298 334 
         FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"happy", "feedback"=>"Take 2"})
 
@@ -306,11 +342,11 @@ describe Agents::HumanTaskAgent do
306 342 
       @checker.send :review_hits
307 343
308 344 
       assignments.all? {|a| a.approved == true }.should be_false
309 
-      @checker.memory[:hits].should == { :"JH3132836336DHG" => @event.id }
345 
+      @checker.memory[:hits].should == { :"JH3132836336DHG" => { :event_id => @event.id } }
310 346 
     end
311 347
312 348 
     it "should create events when all assignments are ready" do
313 
-      @checker.memory[:hits] = { :"JH3132836336DHG" => @event.id }
349 
+      @checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
314 350 
       mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
315 351 
       assignments = [
316 352 
         FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"neutral", "feedback"=>""}),
 
@@ -337,8 +373,8 @@ describe Agents::HumanTaskAgent do
337 373
338 374 
     describe "taking majority votes" do
339 375 
       before do
340 
-        @checker.options[:take_majority] = "true"
341 
-        @checker.memory[:hits] = { :"JH3132836336DHG" => @event.id }
376 
+        @checker.options[:combination_mode] = "take_majority"
377 
+        @checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
342 378 
         mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
343 379 
       end
344 380 
 
@@ -386,6 +422,10 @@ describe Agents::HumanTaskAgent do
386 422 
       end
387 423
388 424 
       it "should also provide an average answer when all questions are numeric" do
425 
+        # it should accept 'take_majority': 'true' as well for legacy support.  Demonstrating that here.
426 
+        @checker.options.delete :combination_mode
427 
+        @checker.options[:take_majority] = "true"
428 
+
389 429 
         @checker.options[:hit][:questions] = [
390 430 
           {
391 431 
             :type => "selection",
 
@@ -435,5 +475,118 @@ describe Agents::HumanTaskAgent do
435 475 
         @checker.memory[:hits].should == {}
436 476 
       end
437 477 
     end
478 
+
479 
+    describe "creating and reviewing polls" do
480 
+      before do
481 
+        @checker.options[:combination_mode] = "poll"
482 
+        @checker.options[:poll_options] = {
483 
+          :title => "Hi!",
484 
+          :instructions => "hello!",
485 
+          :assignments => 2,
486 
+          :row_template => "This is <.sentiment>"
487 
+        }
488 
+        @event.save!
489 
+        mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
490 
+      end
491 
+
492 
+      it "creates a poll using the row_template, message, and correct number of assignments" do
493 
+        @checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
494 
+
495 
+        # Mock out the HIT's submitted assignments.
496 
+        assignments = [
497 
+          FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"sad",     "feedback"=>"This is my feedback 1"}),
498 
+          FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"neutral", "feedback"=>"This is my feedback 2"}),
499 
+          FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"happy",   "feedback"=>"This is my feedback 3"}),
500 
+          FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"happy",   "feedback"=>"This is my feedback 4"})
501 
+        ]
502 
+        hit = FakeHit.new(:max_assignments => 4, :assignments => assignments)
503 
+        mock(RTurk::Hit).new("JH3132836336DHG") { hit }
504 
+
505 
+        @checker.memory[:hits][:"JH3132836336DHG"].should be_present
506 
+
507 
+        # Setup mocks for HIT creation
508 
+
509 
+        question_form = nil
510 
+        hitInterface = OpenStruct.new
511 
+        hitInterface.id = "JH39AA63836DH12345"
512 
+        mock(hitInterface).question_form(instance_of Agents::HumanTaskAgent::AgentQuestionForm) { |agent_question_form_instance| question_form = agent_question_form_instance }
513 
+        mock(RTurk::Hit).create(:title => "Hi!").yields(hitInterface) { hitInterface }
514 
+
515 
+        # And finally, the test.
516 
+
517 
+        lambda {
518 
+          @checker.send :review_hits
519 
+        }.should change { Event.count }.by(0) # it does not emit an event until all poll results are in
520 
+
521 
+        # it approves the existing assignments
522 
+
523 
+        assignments.all? {|a| a.approved == true }.should be_true
524 
+        hit.should be_disposed
525 
+
526 
+        # it creates a new HIT for the poll
527 
+
528 
+        hitInterface.max_assignments.should == @checker.options[:poll_options][:assignments]
529 
+        hitInterface.description.should == @checker.options[:poll_options][:instructions]
530 
+
531 
+        xml = question_form.to_xml
532 
+        xml.should include("<Text>This is happy</Text>")
533 
+        xml.should include("<Text>This is neutral</Text>")
534 
+        xml.should include("<Text>This is sad</Text>")
535 
+
536 
+        @checker.save
537 
+        @checker.reload
538 
+        @checker.memory[:hits][:"JH3132836336DHG"].should_not be_present
539 
+        @checker.memory[:hits][:"JH39AA63836DH12345"].should be_present
540 
+        @checker.memory[:hits][:"JH39AA63836DH12345"][:event_id].should == @event.id
541 
+        @checker.memory[:hits][:"JH39AA63836DH12345"][:type].should == :poll
542 
+        @checker.memory[:hits][:"JH39AA63836DH12345"][:original_hit].should == "JH3132836336DHG"
543 
+        @checker.memory[:hits][:"JH39AA63836DH12345"][:answers].length.should == 4
544 
+      end
545 
+
546 
+      it "emits an event when all poll results are in, containing the data from the best answer, plus all others" do
547 
+        original_answers = [
548 
+          {:sentiment => "sad",     :feedback => "This is my feedback 1"},
549 
+          {:sentiment => "neutral", :feedback => "This is my feedback 2"},
550 
+          {:sentiment => "happy",   :feedback => "This is my feedback 3"},
551 
+          {:sentiment => "happy",   :feedback => "This is my feedback 4"}
552 
+        ]
553 
+
554 
+        @checker.memory[:hits] = {
555 
+          :JH39AA63836DH12345 => {
556 
+            :type => :poll,
557 
+            :original_hit => "JH3132836336DHG",
558 
+            :answers => original_answers,
559 
+            :event_id => 345
560 
+          }
561 
+        }
562 
+
563 
+        # Mock out the HIT's submitted assignments.
564 
+        assignments = [
565 
+          FakeAssignment.new(:status => "Submitted", :answers => {"1" => "2", "2" => "5", "3" => "3", "4" => "2"}),
566 
+          FakeAssignment.new(:status => "Submitted", :answers => {"1" => "3", "2" => "4", "3" => "1", "4" => "4"})
567 
+        ]
568 
+        hit = FakeHit.new(:max_assignments => 2, :assignments => assignments)
569 
+        mock(RTurk::Hit).new("JH39AA63836DH12345") { hit }
570 
+
571 
+        @checker.memory[:hits][:"JH39AA63836DH12345"].should be_present
572 
+
573 
+        lambda {
574 
+          @checker.send :review_hits
575 
+        }.should change { Event.count }.by(1)
576 
+
577 
+        # It emits an event
578 
+
579 
+        @checker.events.last.payload[:answers].should == original_answers
580 
+        @checker.events.last.payload[:poll].should == [{:"1" => "2", :"2" => "5", :"3" => "3", :"4" => "2"}, {:"1" => "3", :"2" => "4", :"3" => "1", :"4" => "4"}]
581 
+        @checker.events.last.payload[:best_answer].should == {:sentiment => "neutral", :feedback => "This is my feedback 2"}
582 
+
583 
+        # it approves the existing assignments
584 
+
585 
+        assignments.all? {|a| a.approved == true }.should be_true
586 
+        hit.should be_disposed
587 
+
588 
+        @checker.memory[:hits].should be_empty
589 
+      end
590 
+    end
438 591 
   end
439 592 
 end