Home Explore Help
Register Sign In
J1X
/
huginn
Watch 2
Star 0
Fork 0

Browse Source
Merge remote-tracking branch 'huginn/master' into chef-solo

Conflicts:
deployment/site-cookbooks/huginn_production/recipes/default.rb

Dominik Sander 11 years ago
parent
8ef3b8ad6e fa9d550194
commit
e7fe019195
13 changed files with 526 additions and 94 deletions
Show Diff Stats
  1. 12 0
      .env.example
  2. 2 1
      CHANGES.md
  3. 1 1
      README.md
  4. 1 1
      app/models/agent.rb
  5. 68 10
      app/models/agents/post_agent.rb
  6. 111 0
      app/models/agents/shell_command_agent.rb
  7. 74 60
      app/models/agents/website_agent.rb
  8. 1 1
      bin/schedule.rb
  9. 4 4
      deployment/site-cookbooks/huginn_development/recipes/default.rb
  10. 2 2
      lib/utils.rb
  11. 127 14
      spec/models/agents/post_agent_spec.rb
  12. 99 0
      spec/models/agents/shell_command_agent_spec.rb
  13. 24 0
      spec/models/agents/website_agent_spec.rb

+ 12 - 0
.env.example
View File 

@@ -78,6 +78,18 @@ AWS_ACCESS_KEY="your aws access key"
78 78 
 # Set AWS_SANDBOX to true if you're developing Huginn code.
79 79 
 AWS_SANDBOX=false
80 80
81 
+########################
82 
+#   Various Settings   #
83 
+########################
84 
+
85 
+# Allow JSONPath eval expresions. i.e., $..price[?(@ < 20)]
86 
+# You should not allow this on a shared Huginn box because it is not secure.
87 
+ALLOW_JSONPATH_EVAL=false
88 
+
89 
+# Enable this setting to allow insecure Agents like the ShellCommandAgent.  Only do this
90 
+# when you trust everyone using your Huginn installation.
91 
+ENABLE_INSECURE_AGENTS=false
92 
+
81 93 
 # Use Graphviz for generating diagrams instead of using Google Chart
82 94 
 # Tools.  Specify a dot(1) command path built with SVG support
83 95 
 # enabled.

+ 2 - 1
CHANGES.md
View File 

@@ -1,6 +1,7 @@
1 1 
 # Changes
2 2
3 
-* 0.4 (April 10, 2014) - WebHooksController has been renamed to WebRequestsController and all HTTP verbs are now accepted and passed through to Agents' #receive_web_request method. The new DataOutputAgent returns JSON or RSS feeds of incoming Events via external web request.
3 
+* 0.5 (April 20, 2014) - Tons of new additions! FtpsiteAgent; WebsiteAgent has xpath, multiple URL, and encoding support; regexp extractions in EventFormattingAgent; PostAgent takes default params and headers, and can make GET requests; local Graphviz support; ShellCommandAgent; BasecampAgent; HipchatAgent; and lots of bug fixes!
4 
+* 0.4 (April 10, 2014) - WebHooksController has been renamed to WebRequestsController and all HTTP verbs are now accepted and passed through to Agents' #receive\_web\_request method. The new DataOutputAgent returns JSON or RSS feeds of incoming Events via external web request.  [Documentation is on the wiki.](https://github.com/cantino/huginn/wiki/Creating-a-new-agent#receiving-web-requests).
4 5 
 * 0.31 (Jan 2, 2014)   - Agents now have an optional keep\_events\_for option that is propagated to created events' expires\_at field, and they update their events' expires\_at fields on change.
5 6 
 * 0.3 (Jan 1, 2014)    - Remove symbolization of memory, options, and payloads; convert memory, options, and payloads to JSON from YAML.  Migration will perform conversion and adjust tables to be UTF-8.  Recommend making a DB backup before migrating.
6 7 
 * 0.2 (Nov 6, 2013)    - PeakDetectorAgent now uses `window_duration_in_days` and `min_peak_spacing_in_days`.  Additionally, peaks trigger when the time series rises over the standard deviation multiple, not after it starts to fall.

+ 1 - 1
README.md
View File 

@@ -24,7 +24,7 @@ Follow [@tectonic](https://twitter.com/tectonic) for updates as Huginn evolves,
24 24
25 25 
 ### We need your help!
26 26
27 
-Want to help with Huginn?  Try tackling [issues tagged with #help-wanted](https://github.com/cantino/huginn/issues?direction=desc&labels=help-wanted&page=1&sort=created&state=open).
27 
+Want to help with Huginn?  All contributions are encouraged!  You could make UI improvements, add new Agents, write documentation and tutorials, or try tackling [issues tagged with #help-wanted](https://github.com/cantino/huginn/issues?direction=desc&labels=help-wanted&page=1&sort=created&state=open).
28 28
29 29 
 ## Examples
30 30

+ 1 - 1
app/models/agent.rb
View File 

@@ -16,7 +16,7 @@ class Agent < ActiveRecord::Base
16 16
17 17 
   load_types_in "Agents"
18 18
19 
-  SCHEDULES = %w[every_2m every_5m every_10m every_30m every_1h every_2h every_5h every_12h every_1d every_2d every_7d
19 
+  SCHEDULES = %w[every_1m every_2m every_5m every_10m every_30m every_1h every_2h every_5h every_12h every_1d every_2d every_7d
20 20 
                  midnight 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am noon 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm never]
21 21
22 22 
   EVENT_RETENTION_SCHEDULES = [["Forever", 0], ["1 day", 1], *([2, 3, 4, 5, 7, 14, 21, 30, 45, 90, 180, 365].map {|n| ["#{n} days", n] })]

+ 68 - 10
app/models/agents/post_agent.rb
View File 

@@ -1,10 +1,15 @@
1 1 
 module Agents
2 2 
   class PostAgent < Agent
3 
-    cannot_be_scheduled!
4 3 
     cannot_create_events!
5 4
5 
+    default_schedule "never"
6 
+
6 7 
     description <<-MD
7 
-       Post Agent receives events from other agents and send those events as the contents of a post request to a specified url. `post_url` field must specify where you would like to receive post requests and do not forget to include URI scheme (`http` or `https`)
8 
+      A PostAgent receives events from other agents (or runs periodically), merges those events with the contents of `payload`, and sends the results as POST (or GET) requests to a specified url.
9 
+
10 
+      The `post_url` field must specify where you would like to send requests. Please include the URI scheme (`http` or `https`).
11 
+
12 
+      The `headers` field is optional.  When present, it should be a hash of headers to send with the request.
8 13 
     MD
9 14
10 15 
     event_description "Does not produce events."
 
@@ -12,7 +17,12 @@ module Agents
12 17 
     def default_options
13 18 
       {
14 19 
         'post_url' => "http://www.example.com",
15 
-        'expected_receive_period_in_days' => 1
20 
+        'expected_receive_period_in_days' => 1,
21 
+        'method' => 'post',
22 
+        'payload' => {
23 
+          'key' => 'value'
24 
+        },
25 
+        'headers' => {}
16 26 
       }
17 27 
     end
18 28 
 
@@ -20,23 +30,71 @@ module Agents
20 30 
       last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
21 31 
     end
22 32
33 
+    def method
34 
+      (options['method'].presence || 'post').to_s.downcase
35 
+    end
36 
+
37 
+    def headers
38 
+      options['headers'].presence || {}
39 
+    end
40 
+
23 41 
     def validate_options
24 42 
       unless options['post_url'].present? && options['expected_receive_period_in_days'].present?
25 43 
         errors.add(:base, "post_url and expected_receive_period_in_days are required fields")
26 44 
       end
27 
-    end
28 45
29 
-    def post_event(uri, event)
30 
-      req = Net::HTTP::Post.new(uri.request_uri)
31 
-      req.form_data = event
32 
-      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
46 
+      if options['payload'].present? && !options['payload'].is_a?(Hash)
47 
+        errors.add(:base, "if provided, payload must be a hash")
48 
+      end
49 
+
50 
+      unless %w[post get].include?(method)
51 
+        errors.add(:base, "method must be 'post' or 'get'")
52 
+      end
53 
+
54 
+      unless headers.is_a?(Hash)
55 
+        errors.add(:base, "if provided, headers must be a hash")
56 
+      end
33 57 
     end
34 58
35 59 
     def receive(incoming_events)
36 60 
       incoming_events.each do |event|
37 
-        uri = URI options[:post_url]
38 
-        post_event uri, event.payload
61 
+        handle (options['payload'].presence || {}).merge(event.payload)
39 62 
       end
40 63 
     end
64 
+
65 
+    def check
66 
+      handle options['payload'].presence || {}
67 
+    end
68 
+
69 
+    def generate_uri(params = nil)
70 
+      uri = URI options[:post_url]
71 
+      uri.query = URI.encode_www_form(Hash[URI.decode_www_form(uri.query || '')].merge(params)) if params
72 
+      uri
73 
+    end
74 
+
75 
+    private
76 
+
77 
+    def handle(data)
78 
+      if method == 'post'
79 
+        post_data(data)
80 
+      elsif method == 'get'
81 
+        get_data(data)
82 
+      else
83 
+        error "Invalid method '#{method}'"
84 
+      end
85 
+    end
86 
+
87 
+    def post_data(data)
88 
+      uri = generate_uri
89 
+      req = Net::HTTP::Post.new(uri.request_uri, headers)
90 
+      req.form_data = data
91 
+      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
92 
+    end
93 
+
94 
+    def get_data(data)
95 
+      uri = generate_uri(data)
96 
+      req = Net::HTTP::Get.new(uri.request_uri, headers)
97 
+      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
98 
+    end
41 99 
   end
42 100 
 end

+ 111 - 0
app/models/agents/shell_command_agent.rb
View File 

@@ -0,0 +1,111 @@
1 
+require 'open3'
2 
+
3 
+module Agents
4 
+  class ShellCommandAgent < Agent
5 
+    default_schedule "never"
6 
+
7 
+    def self.should_run?
8 
+      ENV['ENABLE_INSECURE_AGENTS'] == "true"
9 
+    end
10 
+
11 
+    description <<-MD
12 
+      The ShellCommandAgent can execute commands on your local system, returning the output.
13 
+
14 
+      `command` specifies the command to be executed, and `path` will tell ShellCommandAgent in what directory to run this command.
15 
+
16 
+      `expected_update_period_in_days` is used to determine if the Agent is working.
17 
+
18 
+      ShellCommandAgent can also act upon received events. These events may contain their own `path` and `command` values. If they do not, ShellCommandAgent will use the configured options. For this reason, please specify defaults even if you are planning to have this Agent to respond to events.
19 
+
20 
+      The resulting event will contain the `command` which was executed, the `path` it was executed under, the `exit_status` of the command, the `errors`, and the actual `output`. ShellCommandAgent will not log an error if the result implies that something went wrong.
21 
+
22 
+      *Warning*: This type of Agent runs arbitrary commands on your system, #{Agents::ShellCommandAgent.should_run? ? "but is **currently enabled**" : "and is **currently disabled**"}.
23 
+      Only enable this Agent if you trust everyone using your Huginn installation.
24 
+      You can enable this Agent in your .env file by setting `ENABLE_INSECURE_AGENTS` to `true`.
25 
+    MD
26 
+
27 
+    event_description <<-MD
28 
+    Events look like this:
29 
+
30 
+      {
31 
+        'command' => 'pwd',
32 
+        'path' => '/home/Huginn',
33 
+        'exit_status' => '0',
34 
+        'errors' => '',
35 
+        'output' => '/home/Huginn'
36 
+      }
37 
+    MD
38 
+
39 
+    def default_options
40 
+      {
41 
+          'path' => "/",
42 
+          'command' => "pwd",
43 
+          'expected_update_period_in_days' => 1
44 
+      }
45 
+    end
46 
+
47 
+    def validate_options
48 
+      unless options['path'].present? && options['command'].present? && options['expected_update_period_in_days'].present?
49 
+        errors.add(:base, "The path, command, and expected_update_period_in_days fields are all required.")
50 
+      end
51 
+
52 
+      unless File.directory?(options['path'])
53 
+        errors.add(:base, "#{options['path']} is not a real directory.")
54 
+      end
55 
+    end
56 
+
57 
+    def working?
58 
+      Agents::ShellCommandAgent.should_run? && event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
59 
+    end
60 
+
61 
+    def receive(incoming_events)
62 
+      incoming_events.each do |event|
63 
+        handle(event.payload, event)
64 
+      end
65 
+    end
66 
+
67 
+    def check
68 
+      handle(options)
69 
+    end
70 
+
71 
+    private
72 
+
73 
+    def handle(opts = options, event = nil)
74 
+      if Agents::ShellCommandAgent.should_run?
75 
+        command = opts['command'] || options['command']
76 
+        path = opts['path'] || options['path']
77 
+
78 
+        result, errors, exit_status = run_command(path, command)
79 
+
80 
+        vals = {"command" => command, "path" => path, "exit_status" => exit_status, "errors" => errors, "output" => result}
81 
+        created_event = create_event :payload => vals
82 
+
83 
+        log("Ran '#{command}' under '#{path}'", :outbound_event => created_event, :inbound_event => event)
84 
+      else
85 
+        log("Unable to run because insecure agents are not enabled.  Edit ENABLE_INSECURE_AGENTS in the Huginn .env configuration.")
86 
+      end
87 
+    end
88 
+
89 
+    def run_command(path, command)
90 
+      result = nil
91 
+      errors = nil
92 
+      exit_status = nil
93 
+
94 
+      Dir.chdir(path){
95 
+        begin
96 
+          stdin, stdout, stderr, wait_thr = Open3.popen3(command)
97 
+          exit_status = wait_thr.value.to_i
98 
+          result = stdout.gets(nil)
99 
+          errors = stderr.gets(nil)
100 
+        rescue Exception => e
101 
+          errors = e.to_s
102 
+        end
103 
+      }
104 
+
105 
+      result = result.to_s.strip
106 
+      errors = errors.to_s.strip
107 
+
108 
+      [result, errors, exit_status]
109 
+    end
110 
+  end
111 
+end

+ 74 - 60
app/models/agents/website_agent.rb
View File 

@@ -16,6 +16,8 @@ module Agents
16 16
17 17 
       Specify a `url` and select a `mode` for when to create Events based on the scraped data, either `all` or `on_change`.
18 18
19 
+      `url` can be a single url, or an array of urls (for example, for multiple pages with the exact same structure but different content to scrape)
20 
+
19 21 
       The `type` value can be `xml`, `html`, or `json`.
20 22
21 23 
       To tell the Agent how to parse the content, specify `extract` as a hash with keys naming the extractions and values of hashes.
 
@@ -107,85 +109,97 @@ module Agents
107 109 
       log "Fetching #{options['url']}"
108 110 
       request_opts = { :followlocation => true }
109 111 
       request_opts[:userpwd] = options['basic_auth'] if options['basic_auth'].present?
110 
-      request = Typhoeus::Request.new(options['url'], request_opts)
111 112
112 
-      request.on_failure do |response|
113 
-        error "Failed: #{response.inspect}"
113 
+      requests = []
114 
+
115 
+      if options['url'].kind_of?(Array)
116 
+        options['url'].each do |url|
117 
+           requests.push(Typhoeus::Request.new(url, request_opts))
118 
+        end
119 
+      else
120 
+        requests.push(Typhoeus::Request.new(options['url'], request_opts))
114 121 
       end
115 122
116 
-      request.on_success do |response|
117 
-        body = response.body
118 
-        if (encoding = options['force_encoding']).present?
119 
-          body = body.encode(Encoding::UTF_8, encoding)
123 
+      requests.each do |request|
124 
+        request.on_failure do |response|
125 
+          error "Failed: #{response.inspect}"
120 126 
         end
121 
-        doc = parse(body)
122 127
123 
-        if extract_full_json?
124 
-          if store_payload!(previous_payloads(1), doc)
125 
-            log "Storing new result for '#{name}': #{doc.inspect}"
126 
-            create_event :payload => doc
128 
+        request.on_success do |response|
129 
+          body = response.body
130 
+          if (encoding = options['force_encoding']).present?
131 
+            body = body.encode(Encoding::UTF_8, encoding)
127 132 
           end
128 
-        else
129 
-          output = {}
130 
-          options['extract'].each do |name, extraction_details|
131 
-            if extraction_type == "json"
132 
-              result = Utils.values_at(doc, extraction_details['path'])
133 
-              log "Extracting #{extraction_type} at #{extraction_details['path']}: #{result}"
134 
-            else
135 
-              case
136 
-              when css = extraction_details['css']
137 
-                nodes = doc.css(css)
138 
-              when xpath = extraction_details['xpath']
139 
-                nodes = doc.xpath(xpath)
133 
+          doc = parse(body)
134 
+
135 
+          if extract_full_json?
136 
+            if store_payload!(previous_payloads(1), doc)
137 
+              log "Storing new result for '#{name}': #{doc.inspect}"
138 
+              create_event :payload => doc
139 
+            end
140 
+          else
141 
+            output = {}
142 
+            options['extract'].each do |name, extraction_details|
143 
+              if extraction_type == "json"
144 
+                result = Utils.values_at(doc, extraction_details['path'])
145 
+                log "Extracting #{extraction_type} at #{extraction_details['path']}: #{result}"
140 146 
               else
141 
-                error "'css' or 'xpath' is required for HTML or XML extraction"
142 
-                return
143 
-              end
144 
-              unless Nokogiri::XML::NodeSet === nodes
145 
-                error "The result of HTML/XML extraction was not a NodeSet"
146 
-                return
147 
-              end
148 
-              result = nodes.map { |node|
149 
-                if extraction_details['attr']
150 
-                  node.attr(extraction_details['attr'])
151 
-                elsif extraction_details['text']
152 
-                  node.text()
147 
+                case
148 
+                when css = extraction_details['css']
149 
+                  nodes = doc.css(css)
150 
+                when xpath = extraction_details['xpath']
151 
+                  nodes = doc.xpath(xpath)
153 152 
                 else
154 
-                  error "'attr' or 'text' is required on HTML or XML extraction patterns"
153 
+                  error "'css' or 'xpath' is required for HTML or XML extraction"
155 154 
                   return
156 155 
                 end
157 
-              }
158 
-              log "Extracting #{extraction_type} at #{xpath || css}: #{result}"
156 
+                unless Nokogiri::XML::NodeSet === nodes
157 
+                  error "The result of HTML/XML extraction was not a NodeSet"
158 
+                  return
159 
+                end
160 
+                result = nodes.map { |node|
161 
+                  if extraction_details['attr']
162 
+                    node.attr(extraction_details['attr'])
163 
+                  elsif extraction_details['text']
164 
+                    node.text()
165 
+                  else
166 
+                    error "'attr' or 'text' is required on HTML or XML extraction patterns"
167 
+                    return
168 
+                  end
169 
+                }
170 
+                log "Extracting #{extraction_type} at #{xpath || css}: #{result}"
171 
+              end
172 
+              output[name] = result
159 173 
             end
160 
-            output[name] = result
161 
-          end
162 174
163 
-          num_unique_lengths = options['extract'].keys.map { |name| output[name].length }.uniq
175 
+            num_unique_lengths = options['extract'].keys.map { |name| output[name].length }.uniq
164 176
165 
-          if num_unique_lengths.length != 1
166 
-            error "Got an uneven number of matches for #{options['name']}: #{options['extract'].inspect}"
167 
-            return
168 
-          end
169 
-
170 
-          old_events = previous_payloads num_unique_lengths.first
171 
-          num_unique_lengths.first.times do |index|
172 
-            result = {}
173 
-            options['extract'].keys.each do |name|
174 
-              result[name] = output[name][index]
175 
-              if name.to_s == 'url'
176 
-                result[name] = URI.join(options['url'], result[name]).to_s if (result[name] =~ URI::DEFAULT_PARSER.regexp[:ABS_URI]).nil?
177 
-              end
177 
+            if num_unique_lengths.length != 1
178 
+              error "Got an uneven number of matches for #{options['name']}: #{options['extract'].inspect}"
179 
+              return
178 180 
             end
181 
+
182 
+            old_events = previous_payloads num_unique_lengths.first
183 
+            num_unique_lengths.first.times do |index|
184 
+              result = {}
185 
+              options['extract'].keys.each do |name|
186 
+                result[name] = output[name][index]
187 
+                if name.to_s == 'url'
188 
+                  result[name] = URI.join(options['url'], result[name]).to_s if (result[name] =~ URI::DEFAULT_PARSER.regexp[:ABS_URI]).nil?
189 
+                end
190 
+              end
179 191
180 
-            if store_payload!(old_events, result)
181 
-              log "Storing new parsed result for '#{name}': #{result.inspect}"
182 
-              create_event :payload => result
192 
+              if store_payload!(old_events, result)
193 
+                log "Storing new parsed result for '#{name}': #{result.inspect}"
194 
+                create_event :payload => result
195 
+              end
183 196 
             end
184 197 
           end
185 198 
         end
199 
+
200 
+        hydra.queue request
201 
+        hydra.run
186 202 
       end
187 
-      hydra.queue request
188 
-      hydra.run
189 203 
     end
190 204
191 205 
     private

+ 1 - 1
bin/schedule.rb
View File 

@@ -64,7 +64,7 @@ class HuginnScheduler
64 64
65 65 
     # Schedule repeating events.
66 66
67 
-    %w[2m 5m 10m 30m 1h 2h 5h 12h 1d 2d 7d].each do |schedule|
67 
+    %w[1m 2m 5m 10m 30m 1h 2h 5h 12h 1d 2d 7d].each do |schedule|
68 68 
       rufus_scheduler.every schedule do
69 69 
         run_schedule "every_#{schedule}"
70 70 
       end

+ 4 - 4
deployment/site-cookbooks/huginn_development/recipes/default.rb
View File 

@@ -16,7 +16,7 @@ group "huginn" do
16 16 
   action :create
17 17 
 end
18 18
19 
-%w("ruby1.9.1" "ruby1.9.1-dev" "libxslt-dev" "libxml2-dev" "curl").each do |pkg|
19 
+%w("ruby1.9.1" "ruby1.9.1-dev" "libxslt-dev" "libxml2-dev" "curl" "libmysqlclient-dev").each do |pkg|
20 20 
   package pkg do
21 21 
     action :install
22 22 
   end
 
@@ -49,9 +49,9 @@ bash "huginn dependencies" do
49 49 
     export LC_ALL="en_US.UTF-8"
50 50 
     sudo bundle install
51 51 
     sed s/REPLACE_ME_NOW\!/$(sudo rake secret)/ .env.example > .env
52 
-    sudo rake db:create
53 
-    sudo rake db:migrate
54 
-    sudo rake db:seed
52 
+    sudo bundle exec rake db:create
53 
+    sudo bundle exec rake db:migrate
54 
+    sudo bundle exec rake db:seed
55 55 
     EOH
56 56 
 end
57 57

+ 2 - 2
lib/utils.rb
View File 

@@ -56,7 +56,7 @@ module Utils
56 56 
       escape = false
57 57 
     end
58 58
59 
-    result = JsonPath.new(path, :allow_eval => false).on(data.is_a?(String) ? data : data.to_json)
59 
+    result = JsonPath.new(path, :allow_eval => ENV['ALLOW_JSONPATH_EVAL'] == "true").on(data.is_a?(String) ? data : data.to_json)
60 60 
     if escape
61 61 
       result.map {|r| CGI::escape r }
62 62 
     else
 
@@ -79,4 +79,4 @@ module Utils
79 79 
   def self.pretty_jsonify(thing)
80 80 
     JSON.pretty_generate(thing).gsub('</', '<\/')
81 81 
   end
82 
-end
82 
+end

+ 127 - 14
spec/models/agents/post_agent_spec.rb
View File 

@@ -5,8 +5,11 @@ describe Agents::PostAgent do
5 5 
     @valid_params = {
6 6 
       :name => "somename",
7 7 
       :options => {
8 
-        :post_url => "http://www.example.com",
9 
-        :expected_receive_period_in_days => 1
8 
+        'post_url' => "http://www.example.com",
9 
+        'expected_receive_period_in_days' => 1,
10 
+        'payload' => {
11 
+          'default' => 'value'
12 
+        }
10 13 
       }
11 14 
     }
12 15 
 
@@ -17,28 +20,69 @@ describe Agents::PostAgent do
17 20 
     @event = Event.new
18 21 
     @event.agent = agents(:jane_weather_agent)
19 22 
     @event.payload = {
20 
-      :somekey => "somevalue",
21 
-      :someotherkey => {
22 
-        :somekey => "value"
23 
+      'somekey' => 'somevalue',
24 
+      'someotherkey' => {
25 
+        'somekey' => 'value'
23 26 
       }
24 27 
     }
25 28
26 
-    @sent_messages = []
27 
-    stub.any_instance_of(Agents::PostAgent).post_event { |uri, event| @sent_messages << event }
29 
+    @sent_posts = []
30 
+    @sent_gets = []
31 
+    stub.any_instance_of(Agents::PostAgent).post_data { |data| @sent_posts << data }
32 
+    stub.any_instance_of(Agents::PostAgent).get_data { |data| @sent_gets << data }
28 33 
   end
29 34
30 35 
   describe "#receive" do
31 
-    it "checks if it can handle multiple events" do
36 
+    it "can handle multiple events and merge the payloads with options['payload']" do
32 37 
       event1 = Event.new
33 38 
       event1.agent = agents(:bob_weather_agent)
34 39 
       event1.payload = {
35 
-        :xyz => "value1",
36 
-        :message => "value2"
40 
+        'xyz' => 'value1',
41 
+        'message' => 'value2',
42 
+        'default' => 'value2'
37 43 
       }
38 44
39 45 
       lambda {
40 
-        @checker.receive([@event, event1])
41 
-      }.should change { @sent_messages.length }.by(2)
46 
+        lambda {
47 
+          @checker.receive([@event, event1])
48 
+        }.should change { @sent_posts.length }.by(2)
49 
+      }.should_not change { @sent_gets.length }
50 
+
51 
+      @sent_posts[0].should == @event.payload.merge('default' => 'value')
52 
+      @sent_posts[1].should == event1.payload
53 
+    end
54 
+
55 
+    it "can make GET requests" do
56 
+      @checker.options['method'] = 'get'
57 
+
58 
+      lambda {
59 
+        lambda {
60 
+          @checker.receive([@event])
61 
+        }.should change { @sent_gets.length }.by(1)
62 
+      }.should_not change { @sent_posts.length }
63 
+
64 
+      @sent_gets[0].should == @event.payload.merge('default' => 'value')
65 
+    end
66 
+  end
67 
+
68 
+  describe "#check" do
69 
+    it "sends options['payload'] as a POST request" do
70 
+      lambda {
71 
+        @checker.check
72 
+      }.should change { @sent_posts.length }.by(1)
73 
+
74 
+      @sent_posts[0].should == @checker.options['payload']
75 
+    end
76 
+
77 
+    it "sends options['payload'] as a GET request" do
78 
+      @checker.options['method'] = 'get'
79 
+      lambda {
80 
+        lambda {
81 
+          @checker.check
82 
+        }.should change { @sent_gets.length }.by(1)
83 
+      }.should_not change { @sent_posts.length }
84 
+
85 
+      @sent_gets[0].should == @checker.options['payload']
42 86 
     end
43 87 
   end
44 88 
 
@@ -59,13 +103,82 @@ describe Agents::PostAgent do
59 103 
     end
60 104
61 105 
     it "should validate presence of post_url" do
62 
-      @checker.options[:post_url] = ""
106 
+      @checker.options['post_url'] = ""
63 107 
       @checker.should_not be_valid
64 108 
     end
65 109
66 110 
     it "should validate presence of expected_receive_period_in_days" do
67 
-      @checker.options[:expected_receive_period_in_days] = ""
111 
+      @checker.options['expected_receive_period_in_days'] = ""
68 112 
       @checker.should_not be_valid
69 113 
     end
114 
+
115 
+    it "should validate method as post or get, defaulting to post" do
116 
+      @checker.options['method'] = ""
117 
+      @checker.method.should == "post"
118 
+      @checker.should be_valid
119 
+
120 
+      @checker.options['method'] = "POST"
121 
+      @checker.method.should == "post"
122 
+      @checker.should be_valid
123 
+
124 
+      @checker.options['method'] = "get"
125 
+      @checker.method.should == "get"
126 
+      @checker.should be_valid
127 
+
128 
+      @checker.options['method'] = "wut"
129 
+      @checker.method.should == "wut"
130 
+      @checker.should_not be_valid
131 
+    end
132 
+
133 
+    it "should validate payload as a hash, if present" do
134 
+      @checker.options['payload'] = ""
135 
+      @checker.should be_valid
136 
+
137 
+      @checker.options['payload'] = "hello"
138 
+      @checker.should_not be_valid
139 
+
140 
+      @checker.options['payload'] = ["foo", "bar"]
141 
+      @checker.should_not be_valid
142 
+
143 
+      @checker.options['payload'] = { 'this' => 'that' }
144 
+      @checker.should be_valid
145 
+    end
146 
+
147 
+    it "requires headers to be a hash, if present" do
148 
+      @checker.options['headers'] = [1,2,3]
149 
+      @checker.should_not be_valid
150 
+
151 
+      @checker.options['headers'] = "hello world"
152 
+      @checker.should_not be_valid
153 
+
154 
+      @checker.options['headers'] = ""
155 
+      @checker.should be_valid
156 
+
157 
+      @checker.options['headers'] = {}
158 
+      @checker.should be_valid
159 
+
160 
+      @checker.options['headers'] = { "Authorization" => "foo bar" }
161 
+      @checker.should be_valid
162 
+    end
163 
+  end
164 
+
165 
+  describe "#generate_uri" do
166 
+    it "merges params with any in the post_url" do
167 
+      @checker.options['post_url'] = "http://example.com/a/path?existing_param=existing_value"
168 
+      uri = @checker.generate_uri("some_param" => "some_value", "another_param" => "another_value")
169 
+      uri.request_uri.should == "/a/path?existing_param=existing_value&some_param=some_value&another_param=another_value"
170 
+    end
171 
+
172 
+    it "works fine with urls that do not have a query" do
173 
+      @checker.options['post_url'] = "http://example.com/a/path"
174 
+      uri = @checker.generate_uri("some_param" => "some_value", "another_param" => "another_value")
175 
+      uri.request_uri.should == "/a/path?some_param=some_value&another_param=another_value"
176 
+    end
177 
+
178 
+    it "just returns the post_uri when no params are given" do
179 
+      @checker.options['post_url'] = "http://example.com/a/path?existing_param=existing_value"
180 
+      uri = @checker.generate_uri
181 
+      uri.request_uri.should == "/a/path?existing_param=existing_value"
182 
+    end
70 183 
   end
71 184 
 end

+ 99 - 0
spec/models/agents/shell_command_agent_spec.rb
View File 

@@ -0,0 +1,99 @@
1 
+require 'spec_helper'
2 
+
3 
+describe Agents::ShellCommandAgent do
4 
+  before do
5 
+    @valid_path = Dir.pwd
6 
+
7 
+    @valid_params = {
8 
+        :path  => @valid_path,
9 
+        :command  => "pwd",
10 
+        :expected_update_period_in_days => "1",
11 
+      }
12 
+
13 
+    @checker = Agents::ShellCommandAgent.new(:name => "somename", :options => @valid_params)
14 
+    @checker.user = users(:jane)
15 
+    @checker.save!
16 
+
17 
+    @event = Event.new
18 
+    @event.agent = agents(:jane_weather_agent)
19 
+    @event.payload = {
20 
+      :command => "ls"
21 
+    }
22 
+    @event.save!
23 
+
24 
+    stub(Agents::ShellCommandAgent).should_run? { true }
25 
+  end
26 
+
27 
+  describe "validation" do
28 
+    before do
29 
+      @checker.should be_valid
30 
+    end
31 
+
32 
+    it "should validate presence of necessary fields" do
33 
+      @checker.options[:command] = nil
34 
+      @checker.should_not be_valid
35 
+    end
36 
+
37 
+    it "should validate path" do
38 
+      @checker.options[:path] = 'notarealpath/itreallyisnt'
39 
+      @checker.should_not be_valid
40 
+    end
41 
+
42 
+    it "should validate path" do
43 
+      @checker.options[:path] = '/'
44 
+      @checker.should be_valid
45 
+    end
46 
+  end
47 
+
48 
+  describe "#working?" do
49 
+    it "generating events as scheduled" do
50 
+      stub(@checker).run_command(@valid_path, 'pwd') { ["fake pwd output", "", 0] }
51 
+
52 
+      @checker.should_not be_working
53 
+      @checker.check
54 
+      @checker.reload.should be_working
55 
+      three_days_from_now = 3.days.from_now
56 
+      stub(Time).now { three_days_from_now }
57 
+      @checker.should_not be_working
58 
+    end
59 
+  end
60 
+
61 
+  describe "#check" do
62 
+    before do
63 
+      stub(@checker).run_command(@valid_path, 'pwd') { ["fake pwd output", "", 0] }
64 
+    end
65 
+
66 
+    it "should create an event when checking" do
67 
+      expect { @checker.check }.to change { Event.count }.by(1)
68 
+      Event.last.payload[:path].should == @valid_path
69 
+      Event.last.payload[:command].should == 'pwd'
70 
+      Event.last.payload[:output].should == "fake pwd output"
71 
+    end
72 
+
73 
+    it "does not run when should_run? is false" do
74 
+      stub(Agents::ShellCommandAgent).should_run? { false }
75 
+      expect { @checker.check }.not_to change { Event.count }
76 
+    end
77 
+  end
78 
+
79 
+  describe "#receive" do
80 
+    before do
81 
+      stub(@checker).run_command(@valid_path, @event.payload[:command]) { ["fake ls output", "", 0] }
82 
+    end
83 
+
84 
+    it "creates events" do
85 
+      @checker.receive([@event])
86 
+      Event.last.payload[:path].should == @valid_path
87 
+      Event.last.payload[:command].should == @event.payload[:command]
88 
+      Event.last.payload[:output].should == "fake ls output"
89 
+    end
90 
+
91 
+    it "does not run when should_run? is false" do
92 
+      stub(Agents::ShellCommandAgent).should_run? { false }
93 
+
94 
+      expect {
95 
+        @checker.receive([@event])
96 
+      }.not_to change { Event.count }
97 
+    end
98 
+  end
99 
+end

+ 24 - 0
spec/models/agents/website_agent_spec.rb
View File 

@@ -91,6 +91,30 @@ describe Agents::WebsiteAgent do
91 91 
         @checker.check
92 92 
         @checker.logs.first.message.should =~ /Got an uneven number of matches/
93 93 
       end
94 
+
95 
+      it "should accept an array for url" do
96 
+        @site['url'] = ["http://xkcd.com/1/", "http://xkcd.com/2/"]
97 
+        @checker.options = @site
98 
+        lambda { @checker.save! }.should_not raise_error;
99 
+        lambda { @checker.check }.should_not raise_error;
100 
+      end
101 
+
102 
+      it "should parse events from all urls in array" do
103 
+        lambda {
104 
+          @site['url'] = ["http://xkcd.com/", "http://xkcd.com/"]
105 
+          @site['mode'] = 'all'
106 
+          @checker.options = @site
107 
+          @checker.check
108 
+        }.should change { Event.count }.by(2)
109 
+      end
110 
+
111 
+      it "should follow unique rules when parsing array of urls" do
112 
+        lambda {
113 
+          @site['url'] = ["http://xkcd.com/", "http://xkcd.com/"]
114 
+          @checker.options = @site
115 
+          @checker.check
116 
+        }.should change { Event.count }.by(1)
117 
+      end
94 118 
     end
95 119
96 120 
     describe 'encoding' do