Home Explore Help
Register Sign In
J1X
/
huginn
Watch 2
Star 0
Fork 0

Browse Source
Merge branch 'master' into dsander-rails41

Conflicts:
Gemfile

Andrew Cantino 11 years ago
parent
03a2b09cfa 8767e81985
commit
b688c56c67
22 changed files with 796 additions and 137 deletions
Show Diff Stats
  1. 22 8
      .env.example
  2. 2 1
      CHANGES.md
  3. 2 0
      Gemfile
  4. 7 0
      Gemfile.lock
  5. 2 0
      app/controllers/agents_controller.rb
  6. 0 15
      app/helpers/application_helper.rb
  7. 40 0
      app/helpers/dot_helper.rb
  8. 1 1
      app/models/agent.rb
  9. 68 10
      app/models/agents/post_agent.rb
  10. 111 0
      app/models/agents/shell_command_agent.rb
  11. 16 9
      app/models/agents/trigger_agent.rb
  12. 81 36
      app/models/agents/website_agent.rb
  13. 1 11
      app/views/agents/diagram.html.erb
  14. 1 1
      bin/schedule.rb
  15. 24 27
      deployment/Vagrantfile
  16. 1 0
      deployment/roles/huginn_development.json
  17. 10 3
      deployment/site-cookbooks/huginn_development/recipes/default.rb
  18. 48 0
      spec/helpers/dot_helper_spec.rb
  19. 127 14
      spec/models/agents/post_agent_spec.rb
  20. 99 0
      spec/models/agents/shell_command_agent_spec.rb
  21. 86 0
      spec/models/agents/trigger_agent_spec.rb
  22. 47 1
      spec/models/agents/website_agent_spec.rb

+ 22 - 8
.env.example
View File 

@@ -30,8 +30,18 @@ DATABASE_PASSWORD=""
30 30
31 31 
 # Configure Rails environment.  This should only be needed in production and may cause errors in development.
32 32 
 # RAILS_ENV=production
33 
+
34 
+# Should Rails force all requests to use SSL?
33 35 
 FORCE_SSL=false
34 36
37 
+############################
38 
+#     Allowing Signups     #
39 
+############################
40 
+
41 
+# This invitation code will be required for users to signup with your Huginn installation.
42 
+# You can see its use in user.rb.  PLEASE CHANGE THIS!
43 
+INVITATION_CODE=try-huginn
44 
+
35 45 
 #############################
36 46 
 #    Email Configuration    #
37 47 
 #############################
 
@@ -52,14 +62,7 @@ SMTP_ENABLE_STARTTLS_AUTO=true
52 62
53 63 
 # The address from which system emails will appear to be sent.
54 64 
 EMAIL_FROM_ADDRESS=from_address@gmail.com
55 
-
56 
-############################
57 
-#     Allowing Signups     #
58 
-############################
59 
-
60 
-# This invitation code will be required for users to signup with your Huginn installation.
61 
-# You can see its use in user.rb.
62 
-INVITATION_CODE=try-huginn
65 
+dd
63 66
64 67 
 ###########################
65 68 
 #      Agent Logging      #
 
@@ -83,10 +86,21 @@ AWS_SANDBOX=false
83 86 
 #   Various Settings   #
84 87 
 ########################
85 88
89 
+# Specify the HTTP backend library for Faraday, used in WebsiteAgent.
90 
+# You can change this depending on the performance and stability you
91 
+# need for your service.  Any choice other than "typhoeus",
92 
+# "net_http", or "em_http" should require you to bundle a corresponding
93 
+# gem via Gemfile.
94 
+FARADAY_HTTP_BACKEND=typhoeus
95 
+
86 96 
 # Allow JSONPath eval expresions. i.e., $..price[?(@ < 20)]
87 97 
 # You should not allow this on a shared Huginn box because it is not secure.
88 98 
 ALLOW_JSONPATH_EVAL=false
89 99
100 
+# Enable this setting to allow insecure Agents like the ShellCommandAgent.  Only do this
101 
+# when you trust everyone using your Huginn installation.
102 
+ENABLE_INSECURE_AGENTS=false
103 
+
90 104 
 # Use Graphviz for generating diagrams instead of using Google Chart
91 105 
 # Tools.  Specify a dot(1) command path built with SVG support
92 106 
 # enabled.

+ 2 - 1
CHANGES.md
View File 

@@ -1,6 +1,7 @@
1 1 
 # Changes
2 2
3 
-* 0.4 (April 10, 2014) - WebHooksController has been renamed to WebRequestsController and all HTTP verbs are now accepted and passed through to Agents' #receive_web_request method. The new DataOutputAgent returns JSON or RSS feeds of incoming Events via external web request.
3 
+* 0.5 (April 20, 2014) - Tons of new additions! FtpsiteAgent; WebsiteAgent has xpath, multiple URL, and encoding support; regexp extractions in EventFormattingAgent; PostAgent takes default params and headers, and can make GET requests; local Graphviz support; ShellCommandAgent; BasecampAgent; HipchatAgent; and lots of bug fixes!
4 
+* 0.4 (April 10, 2014) - WebHooksController has been renamed to WebRequestsController and all HTTP verbs are now accepted and passed through to Agents' #receive\_web\_request method. The new DataOutputAgent returns JSON or RSS feeds of incoming Events via external web request.  [Documentation is on the wiki.](https://github.com/cantino/huginn/wiki/Creating-a-new-agent#receiving-web-requests).
4 5 
 * 0.31 (Jan 2, 2014)   - Agents now have an optional keep\_events\_for option that is propagated to created events' expires\_at field, and they update their events' expires\_at fields on change.
5 6 
 * 0.3 (Jan 1, 2014)    - Remove symbolization of memory, options, and payloads; convert memory, options, and payloads to JSON from YAML.  Migration will perform conversion and adjust tables to be UTF-8.  Recommend making a DB backup before migrating.
6 7 
 * 0.2 (Nov 6, 2013)    - PeakDetectorAgent now uses `window_duration_in_days` and `min_peak_spacing_in_days`.  Additionally, peaks trigger when the time series rises over the standard deviation multiple, not after it starts to fall.

+ 2 - 0
Gemfile
View File 

@@ -35,6 +35,8 @@ gem 'geokit', '~> 1.8.4'
35 35 
 gem 'geokit-rails', '~> 2.0.1'
36 36
37 37 
 gem 'kramdown', '~> 1.3.3'
38 
+gem 'faraday', '~> 0.9.0'
39 
+gem 'faraday_middleware'
38 40 
 gem 'typhoeus', '~> 0.6.3'
39 41 
 gem 'nokogiri', '~> 1.6.1'
40 42

+ 7 - 0
Gemfile.lock
View File 

@@ -87,6 +87,7 @@ GEM
87 87 
     diff-lcs (1.2.5)
88 88 
     docile (1.1.3)
89 89 
     dotenv (0.10.0)
90 
+    dotenv-deployment (0.0.2)
90 91 
     dotenv-rails (0.10.0)
91 92 
       dotenv (= 0.10.0)
92 93 
     em-http-request (1.1.2)
 
@@ -107,6 +108,8 @@ GEM
107 108 
     execjs (2.0.2)
108 109 
     faraday (0.9.0)
109 110 
       multipart-post (>= 1.2, < 3)
111 
+    faraday_middleware (0.9.1)
112 
+      faraday (>= 0.7.4, < 0.10)
110 113 
     ffi (1.9.3)
111 114 
     forecast_io (2.0.0)
112 115 
       faraday
 
@@ -317,8 +320,11 @@ DEPENDENCIES
317 320 
   delayed_job_active_record (~> 4.0.0)
318 321 
   delorean
319 322 
   devise (~> 3.2.4)
323 
+  dotenv-deployment
320 324 
   dotenv-rails
321 325 
   em-http-request (~> 1.1.2)
326 
+  faraday (~> 0.9.0)
327 
+  faraday_middleware
322 328 
   forecast_io (~> 2.0.0)
323 329 
   foreman (~> 0.63.0)
324 330 
   geokit (~> 1.8.4)
 
@@ -333,6 +339,7 @@ DEPENDENCIES
333 339 
   nokogiri (~> 1.6.1)
334 340 
   protected_attributes (~> 1.0.7)
335 341 
   pry
342 
+  rack
336 343 
   rails (= 4.1.0)
337 344 
   rr
338 345 
   rspec

+ 2 - 0
app/controllers/agents_controller.rb
View File 

@@ -1,4 +1,6 @@
1 1 
 class AgentsController < ApplicationController
2 
+  include DotHelper
3 
+
2 4 
   def index
3 5 
     @agents = current_user.agents.page(params[:page])
4 6

+ 0 - 15
app/helpers/application_helper.rb
View File 

@@ -14,19 +14,4 @@ module ApplicationHelper
14 14 
       link_to '<span class="label label-warning">No</span>'.html_safe, agent_path(agent, :tab => (agent.recent_error_logs? ? 'logs' : 'details'))
15 15 
     end
16 16 
   end
17 
-
18 
-  def render_dot(dot_format_string)
19 
-    if (command = ENV['USE_GRAPHVIZ_DOT']) &&
20 
-       (svg = IO.popen([command, *%w[-Tsvg -q1 -o/dev/stdout /dev/stdin]], 'w+') { |dot|
21 
-          dot.print dot_format_string
22 
-          dot.close_write
23 
-          dot.read
24 
-        } rescue false)
25 
-      svg.html_safe
26 
-    else
27 
-      tag('img', src: URI('https://chart.googleapis.com/chart').tap { |uri|
28 
-            uri.query = URI.encode_www_form(cht: 'gv', chl: dot_format_string)
29 
-          })
30 
-    end
31 
-  end
32 17 
 end

+ 40 - 0
app/helpers/dot_helper.rb
View File 

@@ -0,0 +1,40 @@
1 
+module DotHelper
2 
+  def render_agents_diagram(agents)
3 
+    if (command = ENV['USE_GRAPHVIZ_DOT']) &&
4 
+       (svg = IO.popen([command, *%w[-Tsvg -q1 -o/dev/stdout /dev/stdin]], 'w+') { |dot|
5 
+          dot.print agents_dot(agents, true)
6 
+          dot.close_write
7 
+          dot.read
8 
+        } rescue false)
9 
+      svg.html_safe
10 
+    else
11 
+      tag('img', src: URI('https://chart.googleapis.com/chart').tap { |uri|
12 
+            uri.query = URI.encode_www_form(cht: 'gv', chl: agents_dot(agents))
13 
+          })
14 
+    end
15 
+  end
16 
+
17 
+  private
18 
+
19 
+  def dot_id(string)
20 
+    # Backslash escaping seems to work for the backslash itself,
21 
+    # despite the DOT language document.
22 
+    '"%s"' % string.gsub(/\\/, "\\\\\\\\").gsub(/"/, "\\\\\"")
23 
+  end
24 
+
25 
+  def agents_dot(agents, rich = false)
26 
+    "digraph foo {".tap { |dot|
27 
+      agents.each.with_index do |agent, index|
28 
+        if rich
29 
+          dot << '%s[URL=%s];' % [dot_id(agent.name), dot_id(agent_path(agent.id))]
30 
+        else
31 
+          dot << '%s;' % dot_id(agent.name)
32 
+        end
33 
+        agent.receivers.each do |receiver|
34 
+          dot << "%s->%s;" % [dot_id(agent.name), dot_id(receiver.name)]
35 
+        end
36 
+      end
37 
+      dot << "}"
38 
+    }
39 
+  end
40 
+end

+ 1 - 1
app/models/agent.rb
View File 

@@ -16,7 +16,7 @@ class Agent < ActiveRecord::Base
16 16
17 17 
   load_types_in "Agents"
18 18
19 
-  SCHEDULES = %w[every_2m every_5m every_10m every_30m every_1h every_2h every_5h every_12h every_1d every_2d every_7d
19 
+  SCHEDULES = %w[every_1m every_2m every_5m every_10m every_30m every_1h every_2h every_5h every_12h every_1d every_2d every_7d
20 20 
                  midnight 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am noon 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm never]
21 21
22 22 
   EVENT_RETENTION_SCHEDULES = [["Forever", 0], ["1 day", 1], *([2, 3, 4, 5, 7, 14, 21, 30, 45, 90, 180, 365].map {|n| ["#{n} days", n] })]

+ 68 - 10
app/models/agents/post_agent.rb
View File 

@@ -1,10 +1,15 @@
1 1 
 module Agents
2 2 
   class PostAgent < Agent
3 
-    cannot_be_scheduled!
4 3 
     cannot_create_events!
5 4
5 
+    default_schedule "never"
6 
+
6 7 
     description <<-MD
7 
-       Post Agent receives events from other agents and send those events as the contents of a post request to a specified url. `post_url` field must specify where you would like to receive post requests and do not forget to include URI scheme (`http` or `https`)
8 
+      A PostAgent receives events from other agents (or runs periodically), merges those events with the contents of `payload`, and sends the results as POST (or GET) requests to a specified url.
9 
+
10 
+      The `post_url` field must specify where you would like to send requests. Please include the URI scheme (`http` or `https`).
11 
+
12 
+      The `headers` field is optional.  When present, it should be a hash of headers to send with the request.
8 13 
     MD
9 14
10 15 
     event_description "Does not produce events."
 
@@ -12,7 +17,12 @@ module Agents
12 17 
     def default_options
13 18 
       {
14 19 
         'post_url' => "http://www.example.com",
15 
-        'expected_receive_period_in_days' => 1
20 
+        'expected_receive_period_in_days' => 1,
21 
+        'method' => 'post',
22 
+        'payload' => {
23 
+          'key' => 'value'
24 
+        },
25 
+        'headers' => {}
16 26 
       }
17 27 
     end
18 28 
 
@@ -20,23 +30,71 @@ module Agents
20 30 
       last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
21 31 
     end
22 32
33 
+    def method
34 
+      (options['method'].presence || 'post').to_s.downcase
35 
+    end
36 
+
37 
+    def headers
38 
+      options['headers'].presence || {}
39 
+    end
40 
+
23 41 
     def validate_options
24 42 
       unless options['post_url'].present? && options['expected_receive_period_in_days'].present?
25 43 
         errors.add(:base, "post_url and expected_receive_period_in_days are required fields")
26 44 
       end
27 
-    end
28 45
29 
-    def post_event(uri, event)
30 
-      req = Net::HTTP::Post.new(uri.request_uri)
31 
-      req.form_data = event
32 
-      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
46 
+      if options['payload'].present? && !options['payload'].is_a?(Hash)
47 
+        errors.add(:base, "if provided, payload must be a hash")
48 
+      end
49 
+
50 
+      unless %w[post get].include?(method)
51 
+        errors.add(:base, "method must be 'post' or 'get'")
52 
+      end
53 
+
54 
+      unless headers.is_a?(Hash)
55 
+        errors.add(:base, "if provided, headers must be a hash")
56 
+      end
33 57 
     end
34 58
35 59 
     def receive(incoming_events)
36 60 
       incoming_events.each do |event|
37 
-        uri = URI options[:post_url]
38 
-        post_event uri, event.payload
61 
+        handle (options['payload'].presence || {}).merge(event.payload)
39 62 
       end
40 63 
     end
64 
+
65 
+    def check
66 
+      handle options['payload'].presence || {}
67 
+    end
68 
+
69 
+    def generate_uri(params = nil)
70 
+      uri = URI options[:post_url]
71 
+      uri.query = URI.encode_www_form(Hash[URI.decode_www_form(uri.query || '')].merge(params)) if params
72 
+      uri
73 
+    end
74 
+
75 
+    private
76 
+
77 
+    def handle(data)
78 
+      if method == 'post'
79 
+        post_data(data)
80 
+      elsif method == 'get'
81 
+        get_data(data)
82 
+      else
83 
+        error "Invalid method '#{method}'"
84 
+      end
85 
+    end
86 
+
87 
+    def post_data(data)
88 
+      uri = generate_uri
89 
+      req = Net::HTTP::Post.new(uri.request_uri, headers)
90 
+      req.form_data = data
91 
+      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
92 
+    end
93 
+
94 
+    def get_data(data)
95 
+      uri = generate_uri(data)
96 
+      req = Net::HTTP::Get.new(uri.request_uri, headers)
97 
+      Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == "https") { |http| http.request(req) }
98 
+    end
41 99 
   end
42 100 
 end

+ 111 - 0
app/models/agents/shell_command_agent.rb
View File 

@@ -0,0 +1,111 @@
1 
+require 'open3'
2 
+
3 
+module Agents
4 
+  class ShellCommandAgent < Agent
5 
+    default_schedule "never"
6 
+
7 
+    def self.should_run?
8 
+      ENV['ENABLE_INSECURE_AGENTS'] == "true"
9 
+    end
10 
+
11 
+    description <<-MD
12 
+      The ShellCommandAgent can execute commands on your local system, returning the output.
13 
+
14 
+      `command` specifies the command to be executed, and `path` will tell ShellCommandAgent in what directory to run this command.
15 
+
16 
+      `expected_update_period_in_days` is used to determine if the Agent is working.
17 
+
18 
+      ShellCommandAgent can also act upon received events. These events may contain their own `path` and `command` values. If they do not, ShellCommandAgent will use the configured options. For this reason, please specify defaults even if you are planning to have this Agent to respond to events.
19 
+
20 
+      The resulting event will contain the `command` which was executed, the `path` it was executed under, the `exit_status` of the command, the `errors`, and the actual `output`. ShellCommandAgent will not log an error if the result implies that something went wrong.
21 
+
22 
+      *Warning*: This type of Agent runs arbitrary commands on your system, #{Agents::ShellCommandAgent.should_run? ? "but is **currently enabled**" : "and is **currently disabled**"}.
23 
+      Only enable this Agent if you trust everyone using your Huginn installation.
24 
+      You can enable this Agent in your .env file by setting `ENABLE_INSECURE_AGENTS` to `true`.
25 
+    MD
26 
+
27 
+    event_description <<-MD
28 
+    Events look like this:
29 
+
30 
+      {
31 
+        'command' => 'pwd',
32 
+        'path' => '/home/Huginn',
33 
+        'exit_status' => '0',
34 
+        'errors' => '',
35 
+        'output' => '/home/Huginn'
36 
+      }
37 
+    MD
38 
+
39 
+    def default_options
40 
+      {
41 
+          'path' => "/",
42 
+          'command' => "pwd",
43 
+          'expected_update_period_in_days' => 1
44 
+      }
45 
+    end
46 
+
47 
+    def validate_options
48 
+      unless options['path'].present? && options['command'].present? && options['expected_update_period_in_days'].present?
49 
+        errors.add(:base, "The path, command, and expected_update_period_in_days fields are all required.")
50 
+      end
51 
+
52 
+      unless File.directory?(options['path'])
53 
+        errors.add(:base, "#{options['path']} is not a real directory.")
54 
+      end
55 
+    end
56 
+
57 
+    def working?
58 
+      Agents::ShellCommandAgent.should_run? && event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
59 
+    end
60 
+
61 
+    def receive(incoming_events)
62 
+      incoming_events.each do |event|
63 
+        handle(event.payload, event)
64 
+      end
65 
+    end
66 
+
67 
+    def check
68 
+      handle(options)
69 
+    end
70 
+
71 
+    private
72 
+
73 
+    def handle(opts = options, event = nil)
74 
+      if Agents::ShellCommandAgent.should_run?
75 
+        command = opts['command'] || options['command']
76 
+        path = opts['path'] || options['path']
77 
+
78 
+        result, errors, exit_status = run_command(path, command)
79 
+
80 
+        vals = {"command" => command, "path" => path, "exit_status" => exit_status, "errors" => errors, "output" => result}
81 
+        created_event = create_event :payload => vals
82 
+
83 
+        log("Ran '#{command}' under '#{path}'", :outbound_event => created_event, :inbound_event => event)
84 
+      else
85 
+        log("Unable to run because insecure agents are not enabled.  Edit ENABLE_INSECURE_AGENTS in the Huginn .env configuration.")
86 
+      end
87 
+    end
88 
+
89 
+    def run_command(path, command)
90 
+      result = nil
91 
+      errors = nil
92 
+      exit_status = nil
93 
+
94 
+      Dir.chdir(path){
95 
+        begin
96 
+          stdin, stdout, stderr, wait_thr = Open3.popen3(command)
97 
+          exit_status = wait_thr.value.to_i
98 
+          result = stdout.gets(nil)
99 
+          errors = stderr.gets(nil)
100 
+        rescue Exception => e
101 
+          errors = e.to_s
102 
+        end
103 
+      }
104 
+
105 
+      result = result.to_s.strip
106 
+      errors = errors.to_s.strip
107 
+
108 
+      [result, errors, exit_status]
109 
+    end
110 
+  end
111 
+end

+ 16 - 9
app/models/agents/trigger_agent.rb
View File 

@@ -11,6 +11,8 @@ module Agents
11 11
12 12 
       The `type` can be one of #{VALID_COMPARISON_TYPES.map { |t| "`#{t}`" }.to_sentence} and compares with the `value`.
13 13
14 
+      The `value` can be a single value or an array of values. In the case of an array, if one or more values match then the rule matches.
15 
+
14 16 
       All rules must match for the Agent to match.  The resulting Event will have a payload message of `message`.  You can include extractions in the message, for example: `I saw a bar of: <foo.bar>`
15 17
16 18 
       Set `expected_receive_period_in_days` to the maximum amount of time that you'd expect to pass between Events being received by this Agent.
 
@@ -49,25 +51,30 @@ module Agents
49 51 
       incoming_events.each do |event|
50 52 
         match = options['rules'].all? do |rule|
51 53 
           value_at_path = Utils.value_at(event['payload'], rule['path'])
52 
-          case rule['type']
54 
+          rule_values = rule['value']
55 
+          rule_values = [rule_values] unless rule_values.is_a?(Array)
56 
+
57 
+          match_found = rule_values.any? do |rule_value|
58 
+            case rule['type']
53 59 
             when "regex"
54 
-              value_at_path.to_s =~ Regexp.new(rule['value'], Regexp::IGNORECASE)
60 
+              value_at_path.to_s =~ Regexp.new(rule_value, Regexp::IGNORECASE)
55 61 
             when "!regex"
56 
-              value_at_path.to_s !~ Regexp.new(rule['value'], Regexp::IGNORECASE)
62 
+              value_at_path.to_s !~ Regexp.new(rule_value, Regexp::IGNORECASE)
57 63 
             when "field>value"
58 
-              value_at_path.to_f > rule['value'].to_f
64 
+              value_at_path.to_f > rule_value.to_f
59 65 
             when "field>=value"
60 
-              value_at_path.to_f >= rule['value'].to_f
66 
+              value_at_path.to_f >= rule_value.to_f
61 67 
             when "field<value"
62 
-              value_at_path.to_f < rule['value'].to_f
68 
+              value_at_path.to_f < rule_value.to_f
63 69 
             when "field<=value"
64 
-              value_at_path.to_f <= rule['value'].to_f
70 
+              value_at_path.to_f <= rule_value.to_f
65 71 
             when "field==value"
66 
-              value_at_path.to_s == rule['value'].to_s
72 
+              value_at_path.to_s == rule_value.to_s
67 73 
             when "field!=value"
68 
-              value_at_path.to_s != rule['value'].to_s
74 
+              value_at_path.to_s != rule_value.to_s
69 75 
             else
70 76 
               raise "Invalid type of #{rule['type']} in TriggerAgent##{id}"
77 
+            end
71 78 
           end
72 79 
         end
73 80

+ 81 - 36
app/models/agents/website_agent.rb
View File 

@@ -1,10 +1,10 @@
1 1 
 require 'nokogiri'
2 
-require 'typhoeus'
2 
+require 'faraday'
3 
+require 'faraday_middleware'
3 4 
 require 'date'
4 5
5 6 
 module Agents
6 7 
   class WebsiteAgent < Agent
7 
-    cannot_receive_events!
8 8
9 9 
     default_schedule "every_12h"
10 10 
 
@@ -22,30 +22,34 @@ module Agents
22 22
23 23 
       To tell the Agent how to parse the content, specify `extract` as a hash with keys naming the extractions and values of hashes.
24 24
25 
-      When parsing HTML or XML, these sub-hashes specify how to extract with either a `css` CSS selector or a `xpath` XPath expression and either `'text': true` or `attr` pointing to an attribute name to grab.  An example:
25 
+      When parsing HTML or XML, these sub-hashes specify how to extract with either a `css` CSS selector or a `xpath` XPath expression and either `"text": true` or `attr` pointing to an attribute name to grab.  An example:
26 26
27 
-          'extract': {
28 
-            'url': { 'css': "#comic img", 'attr': "src" },
29 
-            'title': { 'css': "#comic img", 'attr': "title" },
30 
-            'body_text': { 'css': "div.main", 'text': true }
27 
+          "extract": {
28 
+            "url": { "css": "#comic img", "attr": "src" },
29 
+            "title": { "css": "#comic img", "attr": "title" },
30 
+            "body_text": { "css": "div.main", "text": true }
31 31 
           }
32 32
33 33 
       When parsing JSON, these sub-hashes specify [JSONPaths](http://goessner.net/articles/JsonPath/) to the values that you care about.  For example:
34 34
35 
-          'extract': {
36 
-            'title': { 'path': "results.data[*].title" },
37 
-            'description': { 'path': "results.data[*].description" }
35 
+          "extract": {
36 
+            "title": { "path": "results.data[*].title" },
37 
+            "description": { "path": "results.data[*].description" }
38 38 
           }
39 39
40 40 
       Note that for all of the formats, whatever you extract MUST have the same number of matches for each extractor.  E.g., if you're extracting rows, all extractors must match all rows.  For generating CSS selectors, something like [SelectorGadget](http://selectorgadget.com) may be helpful.
41 41
42 
-      Can be configured to use HTTP basic auth by including the `basic_auth` parameter with `username:password`.
42 
+      Can be configured to use HTTP basic auth by including the `basic_auth` parameter with `"username:password"`, or `["username", "password"]`.
43 43
44 44 
       Set `expected_update_period_in_days` to the maximum amount of time that you'd expect to pass between Events being created by this Agent.  This is only used to set the "working" status.
45 45
46 46 
       Set `uniqueness_look_back` to limit the number of events checked for uniqueness (typically for performance).  This defaults to the larger of #{UNIQUENESS_LOOK_BACK} or #{UNIQUENESS_FACTOR}x the number of detected received results.
47 47
48 48 
       Set `force_encoding` to an encoding name if the website does not return a Content-Type header with a proper charset.
49 
+
50 
+      Set `user_agent` to a custom User-Agent name if the website does not like the default value ("Faraday v#{Faraday::VERSION}").
51 
+
52 
+      The WebsiteAgent can also scrape based on incoming events. It will scrape the url contained in the `url` key of the incoming event payload.
49 53 
     MD
50 54
51 55 
     event_description do
 
@@ -102,30 +106,29 @@ module Agents
102 106 
           errors.add(:base, "force_encoding must be a string")
103 107 
         end
104 108 
       end
105 
-    end
106 
-
107 
-    def check
108 
-      hydra = Typhoeus::Hydra.new
109 
-      log "Fetching #{options['url']}"
110 
-      request_opts = { :followlocation => true }
111 
-      request_opts[:userpwd] = options['basic_auth'] if options['basic_auth'].present?
112 109
113 
-      requests = []
110 
+      if options['user_agent'].present?
111 
+        errors.add(:base, "user_agent must be a string") unless options['user_agent'].is_a?(String)
112 
+      end
114 113
115 
-      if options['url'].kind_of?(Array)
116 
-        options['url'].each do |url|
117 
-           requests.push(Typhoeus::Request.new(url, request_opts))
118 
-        end
119 
-      else
120 
-        requests.push(Typhoeus::Request.new(options['url'], request_opts))
114 
+      begin
115 
+        basic_auth_credentials()
116 
+      rescue => e
117 
+        errors.add(:base, e.message)
121 118 
       end
119 
+    end
122 120
123 
-      requests.each do |request|
124 
-        request.on_failure do |response|
125 
-          error "Failed: #{response.inspect}"
126 
-        end
121 
+    def check
122 
+      check_url options['url']
123 
+    end
124 
+
125 
+    def check_url(in_url)
126 
+      return unless in_url.present?
127 127
128 
-        request.on_success do |response|
128 
+      Array(in_url).each do |url|
129 
+        log "Fetching #{url}"
130 
+        response = faraday.get(url)
131 
+        if response.success?
129 132 
           body = response.body
130 133 
           if (encoding = options['force_encoding']).present?
131 134 
             body = body.encode(Encoding::UTF_8, encoding)
 
@@ -150,7 +153,7 @@ module Agents
150 153 
                 when xpath = extraction_details['xpath']
151 154 
                   nodes = doc.xpath(xpath)
152 155 
                 else
153 
-                  error "'css' or 'xpath' is required for HTML or XML extraction"
156 
+                  error '"css" or "xpath" is required for HTML or XML extraction'
154 157 
                   return
155 158 
                 end
156 159 
                 unless Nokogiri::XML::NodeSet === nodes
 
@@ -163,7 +166,7 @@ module Agents
163 166 
                   elsif extraction_details['text']
164 167 
                     node.text()
165 168 
                   else
166 
-                    error "'attr' or 'text' is required on HTML or XML extraction patterns"
169 
+                    error '"attr" or "text" is required on HTML or XML extraction patterns'
167 170 
                     return
168 171 
                   end
169 172 
                 }
 
@@ -178,14 +181,14 @@ module Agents
178 181 
               error "Got an uneven number of matches for #{options['name']}: #{options['extract'].inspect}"
179 182 
               return
180 183 
             end
181 
-
184 
+
182 185 
             old_events = previous_payloads num_unique_lengths.first
183 186 
             num_unique_lengths.first.times do |index|
184 187 
               result = {}
185 188 
               options['extract'].keys.each do |name|
186 189 
                 result[name] = output[name][index]
187 190 
                 if name.to_s == 'url'
188 
-                  result[name] = URI.join(options['url'], result[name]).to_s if (result[name] =~ URI::DEFAULT_PARSER.regexp[:ABS_URI]).nil?
191 
+                  result[name] = (response.env[:url] + result[name]).to_s
189 192 
                 end
190 193 
               end
191 194 
 
@@ -195,10 +198,16 @@ module Agents
195 198 
               end
196 199 
             end
197 200 
           end
201 
+        else
202 
+          error "Failed: #{response.inspect}"
198 203 
         end
204 
+      end
205 
+    end
199 206
200 
-        hydra.queue request
201 
-        hydra.run
207 
+    def receive(incoming_events)
208 
+      incoming_events.each do |event|
209 
+        url_to_scrape = event.payload['url']
210 
+        check_url(url_to_scrape) if url_to_scrape =~ /^https?:\/\//i
202 211 
       end
203 212 
     end
204 213 
 
@@ -275,5 +284,41 @@ module Agents
275 284 
         false
276 285 
       end
277 286 
     end
287 
+
288 
+    def faraday
289 
+      @faraday ||= Faraday.new { |builder|
290 
+        if (user_agent = options['user_agent']).present?
291 
+          builder.headers[:user_agent] = user_agent
292 
+        end
293 
+
294 
+        builder.use FaradayMiddleware::FollowRedirects
295 
+        builder.request :url_encoded
296 
+        if userinfo = basic_auth_credentials()
297 
+          builder.request :basic_auth, *userinfo
298 
+        end
299 
+
300 
+        case backend = faraday_backend
301 
+        when :typhoeus
302 
+          require 'typhoeus/adapters/faraday'
303 
+        end
304 
+        builder.adapter backend
305 
+      }
306 
+    end
307 
+
308 
+    def faraday_backend
309 
+      ENV.fetch('FARADAY_HTTP_BACKEND', 'typhoeus').to_sym
310 
+    end
311 
+
312 
+    def basic_auth_credentials
313 
+      case value = options['basic_auth']
314 
+      when nil, ''
315 
+        return nil
316 
+      when Array
317 
+        return value if value.size == 2
318 
+      when /:/
319 
+        return value.split(/:/, 2)
320 
+      end
321 
+      raise "bad value for basic_auth: #{value.inspect}"
322 
+    end
278 323 
   end
279 324 
 end

+ 1 - 11
app/views/agents/diagram.html.erb
View File 

@@ -9,17 +9,7 @@
9 9 
       </div>
10 10
11 11 
       <div class='digraph'>
12 
-        <%
13 
-           dot_format_string = "digraph foo {"
14 
-           @agents.each.with_index do |agent, index|
15 
-             dot_format_string += "\"#{agent.name}\";"
16 
-             agent.receivers.each do |receiver|
17 
-               dot_format_string += "\"#{agent.name}\"->\"#{receiver.name}\";"
18 
-             end
19 
-           end
20 
-           dot_format_string = dot_format_string + "}"
21 
-        %>
22 
-        <%= render_dot(dot_format_string) %>
12 
+        <%= render_agents_diagram(@agents) %>
23 13 
       </div>
24 14 
     </div>
25 15 
   </div>

+ 1 - 1
bin/schedule.rb
View File 

@@ -64,7 +64,7 @@ class HuginnScheduler
64 64
65 65 
     # Schedule repeating events.
66 66
67 
-    %w[2m 5m 10m 30m 1h 2h 5h 12h 1d 2d 7d].each do |schedule|
67 
+    %w[1m 2m 5m 10m 30m 1h 2h 5h 12h 1d 2d 7d].each do |schedule|
68 68 
       rufus_scheduler.every schedule do
69 69 
         run_schedule "every_#{schedule}"
70 70 
       end

+ 24 - 27
deployment/Vagrantfile
View File 

@@ -3,37 +3,34 @@
3 3
4 4 
 Vagrant.configure("2") do |config|
5 5 
   config.omnibus.chef_version = :latest
6 
-  config.vm.define :vb do |vb|
7 
-    vb.vm.box = "precise32"
8 
-    vb.vm.box_url = "http://files.vagrantup.com/precise32.box"
9 
-    vb.vm.network :forwarded_port, host: 3000, guest: 3000
10 6
11 
-    vb.vm.provision :chef_solo do |chef|
12 
-      chef.roles_path = "roles"
13 
-      chef.cookbooks_path = ["cookbooks", "site-cookbooks"]
14 
-      chef.add_role("huginn_development")
15 
-    end
7 
+  config.vm.provision :chef_solo do |chef|
8 
+    chef.roles_path = "roles"
9 
+    chef.cookbooks_path = ["cookbooks", "site-cookbooks"]
10 
+    chef.add_role("huginn_development")
11 
+    # chef.add_role("huginn_production")
16 12 
   end
17 13
18 
-  config.vm.define :ec2 do |ec2|
19 
-    ec2.vm.box = "dummy"
20 
-    ec2.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
14 
+  config.vm.provider :virtualbox do |vb, override|
15 
+    override.vm.box = "hashicorp/precise64"
16 
+    override.vm.network :forwarded_port, host: 3000, guest: 3000
17 
+  end
18 
+
19 
+  config.vm.provider :parallels do |prl, override|
20 
+    override.vm.box = "parallels/ubuntu-12.04"
21 
+  end
22 
+
23 
+  config.vm.provider :aws do |aws, override|
24 
+    override.vm.box = "dummy"
25 
+    override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
21 26
22 
-    ec2.vm.provider :aws do |aws, override|
23 
-      aws.access_key_id = ""
24 
-      aws.secret_access_key = ""
25 
-      aws.keypair_name = ""
26 
-      aws.region = "us-east-1"
27 
-      aws.ami = "ami-d0f89fb9"
27 
+    aws.access_key_id = ""
28 
+    aws.secret_access_key = ""
29 
+    aws.keypair_name = ""
30 
+    aws.region = "us-east-1"
31 
+    aws.ami = "ami-d0f89fb9"
28 32
29 
-      override.ssh.username = "ubuntu"
30 
-      override.ssh.private_key_path = ""
31 
-    end
32 
-    ec2.vm.provision :chef_solo do |chef|
33 
-      chef.roles_path = "roles"
34 
-      chef.cookbooks_path = ["cookbooks", "site-cookbooks"]
35 
-      chef.add_role("huginn_production")
36 
-
37 
-    end
33 
+    override.ssh.username = "ubuntu"
34 
+    override.ssh.private_key_path = ""
38 35 
   end
39 36 
 end

+ 1 - 0
deployment/roles/huginn_development.json
View File 

@@ -23,6 +23,7 @@
23 23 
              "recipe[git]",
24 24 
              "recipe[apt]",
25 25 
              "recipe[mysql::server]",
26 
+             "recipe[mysql::client]",
26 27 
              "recipe[nodejs::install_from_binary]",
27 28 
              "recipe[huginn_development]"
28 29 
            ]

+ 10 - 3
deployment/site-cookbooks/huginn_development/recipes/default.rb
View File 

@@ -16,12 +16,19 @@ group "huginn" do
16 16 
   action :create
17 17 
 end
18 18
19 
-%w("ruby1.9.1" "ruby1.9.1-dev" "libxslt-dev" "libxml2-dev" "curl" "libmysqlclient-dev").each do |pkg|
19 
+%w("ruby1.9.1" "ruby1.9.1-dev" "libxslt-dev" "libxml2-dev" "curl" "libmysqlclient-dev" "rubygems").each do |pkg|
20 20 
   package pkg do
21 21 
     action :install
22 22 
   end
23 23 
 end
24 24
25 
+bash "Setting default ruby version to 1.9" do
26 
+  code <<-EOH
27 
+    update-alternatives --set ruby /usr/bin/ruby1.9.1
28 
+    update-alternatives --set gem /usr/bin/gem1.9.1
29 
+  EOH
30 
+end
31 
+
25 32 
 git "/home/huginn/huginn" do
26 33 
   repository 'git://github.com/cantino/huginn.git'
27 34 
   reference 'master'
 
@@ -48,7 +55,7 @@ bash "huginn dependencies" do
48 55 
     export LANG="en_US.UTF-8"
49 56 
     export LC_ALL="en_US.UTF-8"
50 57 
     sudo bundle install
51 
-    sed s/REPLACE_ME_NOW\!/$(sudo rake secret)/ .env.example > .env
58 
+    sed s/REPLACE_ME_NOW\!/$(sudo bundle exec rake secret)/ .env.example > .env
52 59 
     sudo bundle exec rake db:create
53 60 
     sudo bundle exec rake db:migrate
54 61 
     sudo bundle exec rake db:seed
 
@@ -59,6 +66,6 @@ bash "huginn has been installed and will start in a minute" do
59 66 
   user "huginn"
60 67 
   cwd "/home/huginn/huginn"
61 68 
   code <<-EOH
62 
-    sudo foreman start
69 
+    sudo nohup foreman start &
63 70 
     EOH
64 71 
 end

+ 48 - 0
spec/helpers/dot_helper_spec.rb
View File 

@@ -0,0 +1,48 @@
1 
+require 'spec_helper'
2 
+
3 
+describe DotHelper do
4 
+  describe "#dot_id" do
5 
+    it "properly escapes double quotaion and backslash" do
6 
+      dot_id('hello\\"').should == '"hello\\\\\\""'
7 
+    end
8 
+  end
9 
+
10 
+  describe "with example Agents" do
11 
+    class Agents::DotFoo < Agent
12 
+      default_schedule "2pm"
13 
+
14 
+      def check
15 
+        create_event :payload => {}
16 
+      end
17 
+    end
18 
+
19 
+    class Agents::DotBar < Agent
20 
+      cannot_be_scheduled!
21 
+
22 
+      def check
23 
+        create_event :payload => {}
24 
+      end
25 
+    end
26 
+
27 
+    before do
28 
+      stub(Agents::DotFoo).valid_type?("Agents::DotFoo") { true }
29 
+      stub(Agents::DotBar).valid_type?("Agents::DotBar") { true }
30 
+    end
31 
+
32 
+    describe "#agents_dot" do
33 
+      it "generates a DOT script" do
34 
+        @foo = Agents::DotFoo.new(:name => "foo")
35 
+        @foo.user = users(:bob)
36 
+        @foo.save!
37 
+
38 
+        @bar = Agents::DotBar.new(:name => "bar")
39 
+        @bar.user = users(:bob)
40 
+        @bar.sources << @foo
41 
+        @bar.save!
42 
+
43 
+        agents_dot([@foo, @bar]).should == 'digraph foo {"foo";"foo"->"bar";"bar";}'
44 
+        agents_dot([@foo, @bar], true).should == 'digraph foo {"foo"[URL="/agents/%d"];"foo"->"bar";"bar"[URL="/agents/%d"];}' % [@foo.id, @bar.id]
45 
+      end
46 
+    end
47 
+  end
48 
+end

+ 127 - 14
spec/models/agents/post_agent_spec.rb
View File 

@@ -5,8 +5,11 @@ describe Agents::PostAgent do
5 5 
     @valid_params = {
6 6 
       :name => "somename",
7 7 
       :options => {
8 
-        :post_url => "http://www.example.com",
9 
-        :expected_receive_period_in_days => 1
8 
+        'post_url' => "http://www.example.com",
9 
+        'expected_receive_period_in_days' => 1,
10 
+        'payload' => {
11 
+          'default' => 'value'
12 
+        }
10 13 
       }
11 14 
     }
12 15 
 
@@ -17,28 +20,69 @@ describe Agents::PostAgent do
17 20 
     @event = Event.new
18 21 
     @event.agent = agents(:jane_weather_agent)
19 22 
     @event.payload = {
20 
-      :somekey => "somevalue",
21 
-      :someotherkey => {
22 
-        :somekey => "value"
23 
+      'somekey' => 'somevalue',
24 
+      'someotherkey' => {
25 
+        'somekey' => 'value'
23 26 
       }
24 27 
     }
25 28
26 
-    @sent_messages = []
27 
-    stub.any_instance_of(Agents::PostAgent).post_event { |uri, event| @sent_messages << event }
29 
+    @sent_posts = []
30 
+    @sent_gets = []
31 
+    stub.any_instance_of(Agents::PostAgent).post_data { |data| @sent_posts << data }
32 
+    stub.any_instance_of(Agents::PostAgent).get_data { |data| @sent_gets << data }
28 33 
   end
29 34
30 35 
   describe "#receive" do
31 
-    it "checks if it can handle multiple events" do
36 
+    it "can handle multiple events and merge the payloads with options['payload']" do
32 37 
       event1 = Event.new
33 38 
       event1.agent = agents(:bob_weather_agent)
34 39 
       event1.payload = {
35 
-        :xyz => "value1",
36 
-        :message => "value2"
40 
+        'xyz' => 'value1',
41 
+        'message' => 'value2',
42 
+        'default' => 'value2'
37 43 
       }
38 44
39 45 
       lambda {
40 
-        @checker.receive([@event, event1])
41 
-      }.should change { @sent_messages.length }.by(2)
46 
+        lambda {
47 
+          @checker.receive([@event, event1])
48 
+        }.should change { @sent_posts.length }.by(2)
49 
+      }.should_not change { @sent_gets.length }
50 
+
51 
+      @sent_posts[0].should == @event.payload.merge('default' => 'value')
52 
+      @sent_posts[1].should == event1.payload
53 
+    end
54 
+
55 
+    it "can make GET requests" do
56 
+      @checker.options['method'] = 'get'
57 
+
58 
+      lambda {
59 
+        lambda {
60 
+          @checker.receive([@event])
61 
+        }.should change { @sent_gets.length }.by(1)
62 
+      }.should_not change { @sent_posts.length }
63 
+
64 
+      @sent_gets[0].should == @event.payload.merge('default' => 'value')
65 
+    end
66 
+  end
67 
+
68 
+  describe "#check" do
69 
+    it "sends options['payload'] as a POST request" do
70 
+      lambda {
71 
+        @checker.check
72 
+      }.should change { @sent_posts.length }.by(1)
73 
+
74 
+      @sent_posts[0].should == @checker.options['payload']
75 
+    end
76 
+
77 
+    it "sends options['payload'] as a GET request" do
78 
+      @checker.options['method'] = 'get'
79 
+      lambda {
80 
+        lambda {
81 
+          @checker.check
82 
+        }.should change { @sent_gets.length }.by(1)
83 
+      }.should_not change { @sent_posts.length }
84 
+
85 
+      @sent_gets[0].should == @checker.options['payload']
42 86 
     end
43 87 
   end
44 88 
 
@@ -59,13 +103,82 @@ describe Agents::PostAgent do
59 103 
     end
60 104
61 105 
     it "should validate presence of post_url" do
62 
-      @checker.options[:post_url] = ""
106 
+      @checker.options['post_url'] = ""
63 107 
       @checker.should_not be_valid
64 108 
     end
65 109
66 110 
     it "should validate presence of expected_receive_period_in_days" do
67 
-      @checker.options[:expected_receive_period_in_days] = ""
111 
+      @checker.options['expected_receive_period_in_days'] = ""
68 112 
       @checker.should_not be_valid
69 113 
     end
114 
+
115 
+    it "should validate method as post or get, defaulting to post" do
116 
+      @checker.options['method'] = ""
117 
+      @checker.method.should == "post"
118 
+      @checker.should be_valid
119 
+
120 
+      @checker.options['method'] = "POST"
121 
+      @checker.method.should == "post"
122 
+      @checker.should be_valid
123 
+
124 
+      @checker.options['method'] = "get"
125 
+      @checker.method.should == "get"
126 
+      @checker.should be_valid
127 
+
128 
+      @checker.options['method'] = "wut"
129 
+      @checker.method.should == "wut"
130 
+      @checker.should_not be_valid
131 
+    end
132 
+
133 
+    it "should validate payload as a hash, if present" do
134 
+      @checker.options['payload'] = ""
135 
+      @checker.should be_valid
136 
+
137 
+      @checker.options['payload'] = "hello"
138 
+      @checker.should_not be_valid
139 
+
140 
+      @checker.options['payload'] = ["foo", "bar"]
141 
+      @checker.should_not be_valid
142 
+
143 
+      @checker.options['payload'] = { 'this' => 'that' }
144 
+      @checker.should be_valid
145 
+    end
146 
+
147 
+    it "requires headers to be a hash, if present" do
148 
+      @checker.options['headers'] = [1,2,3]
149 
+      @checker.should_not be_valid
150 
+
151 
+      @checker.options['headers'] = "hello world"
152 
+      @checker.should_not be_valid
153 
+
154 
+      @checker.options['headers'] = ""
155 
+      @checker.should be_valid
156 
+
157 
+      @checker.options['headers'] = {}
158 
+      @checker.should be_valid
159 
+
160 
+      @checker.options['headers'] = { "Authorization" => "foo bar" }
161 
+      @checker.should be_valid
162 
+    end
163 
+  end
164 
+
165 
+  describe "#generate_uri" do
166 
+    it "merges params with any in the post_url" do
167 
+      @checker.options['post_url'] = "http://example.com/a/path?existing_param=existing_value"
168 
+      uri = @checker.generate_uri("some_param" => "some_value", "another_param" => "another_value")
169 
+      uri.request_uri.should == "/a/path?existing_param=existing_value&some_param=some_value&another_param=another_value"
170 
+    end
171 
+
172 
+    it "works fine with urls that do not have a query" do
173 
+      @checker.options['post_url'] = "http://example.com/a/path"
174 
+      uri = @checker.generate_uri("some_param" => "some_value", "another_param" => "another_value")
175 
+      uri.request_uri.should == "/a/path?some_param=some_value&another_param=another_value"
176 
+    end
177 
+
178 
+    it "just returns the post_uri when no params are given" do
179 
+      @checker.options['post_url'] = "http://example.com/a/path?existing_param=existing_value"
180 
+      uri = @checker.generate_uri
181 
+      uri.request_uri.should == "/a/path?existing_param=existing_value"
182 
+    end
70 183 
   end
71 184 
 end

+ 99 - 0
spec/models/agents/shell_command_agent_spec.rb
View File 

@@ -0,0 +1,99 @@
1 
+require 'spec_helper'
2 
+
3 
+describe Agents::ShellCommandAgent do
4 
+  before do
5 
+    @valid_path = Dir.pwd
6 
+
7 
+    @valid_params = {
8 
+        :path  => @valid_path,
9 
+        :command  => "pwd",
10 
+        :expected_update_period_in_days => "1",
11 
+      }
12 
+
13 
+    @checker = Agents::ShellCommandAgent.new(:name => "somename", :options => @valid_params)
14 
+    @checker.user = users(:jane)
15 
+    @checker.save!
16 
+
17 
+    @event = Event.new
18 
+    @event.agent = agents(:jane_weather_agent)
19 
+    @event.payload = {
20 
+      :command => "ls"
21 
+    }
22 
+    @event.save!
23 
+
24 
+    stub(Agents::ShellCommandAgent).should_run? { true }
25 
+  end
26 
+
27 
+  describe "validation" do
28 
+    before do
29 
+      @checker.should be_valid
30 
+    end
31 
+
32 
+    it "should validate presence of necessary fields" do
33 
+      @checker.options[:command] = nil
34 
+      @checker.should_not be_valid
35 
+    end
36 
+
37 
+    it "should validate path" do
38 
+      @checker.options[:path] = 'notarealpath/itreallyisnt'
39 
+      @checker.should_not be_valid
40 
+    end
41 
+
42 
+    it "should validate path" do
43 
+      @checker.options[:path] = '/'
44 
+      @checker.should be_valid
45 
+    end
46 
+  end
47 
+
48 
+  describe "#working?" do
49 
+    it "generating events as scheduled" do
50 
+      stub(@checker).run_command(@valid_path, 'pwd') { ["fake pwd output", "", 0] }
51 
+
52 
+      @checker.should_not be_working
53 
+      @checker.check
54 
+      @checker.reload.should be_working
55 
+      three_days_from_now = 3.days.from_now
56 
+      stub(Time).now { three_days_from_now }
57 
+      @checker.should_not be_working
58 
+    end
59 
+  end
60 
+
61 
+  describe "#check" do
62 
+    before do
63 
+      stub(@checker).run_command(@valid_path, 'pwd') { ["fake pwd output", "", 0] }
64 
+    end
65 
+
66 
+    it "should create an event when checking" do
67 
+      expect { @checker.check }.to change { Event.count }.by(1)
68 
+      Event.last.payload[:path].should == @valid_path
69 
+      Event.last.payload[:command].should == 'pwd'
70 
+      Event.last.payload[:output].should == "fake pwd output"
71 
+    end
72 
+
73 
+    it "does not run when should_run? is false" do
74 
+      stub(Agents::ShellCommandAgent).should_run? { false }
75 
+      expect { @checker.check }.not_to change { Event.count }
76 
+    end
77 
+  end
78 
+
79 
+  describe "#receive" do
80 
+    before do
81 
+      stub(@checker).run_command(@valid_path, @event.payload[:command]) { ["fake ls output", "", 0] }
82 
+    end
83 
+
84 
+    it "creates events" do
85 
+      @checker.receive([@event])
86 
+      Event.last.payload[:path].should == @valid_path
87 
+      Event.last.payload[:command].should == @event.payload[:command]
88 
+      Event.last.payload[:output].should == "fake ls output"
89 
+    end
90 
+
91 
+    it "does not run when should_run? is false" do
92 
+      stub(Agents::ShellCommandAgent).should_run? { false }
93 
+
94 
+      expect {
95 
+        @checker.receive([@event])
96 
+      }.not_to change { Event.count }
97 
+    end
98 
+  end
99 
+end

+ 86 - 0
spec/models/agents/trigger_agent_spec.rb
View File 

@@ -71,6 +71,28 @@ describe Agents::TriggerAgent do
71 71 
       }.should change { Event.count }.by(1)
72 72 
     end
73 73
74 
+    it "handles array of regex" do
75 
+      @event.payload['foo']['bar']['baz'] = "a222b"
76 
+      @checker.options['rules'][0] = {
77 
+        'type' => "regex",
78 
+        'value' => ["a\\db", "a\\Wb"],
79 
+        'path' => "foo.bar.baz",
80 
+      }
81 
+      lambda {
82 
+        @checker.receive([@event])
83 
+      }.should_not change { Event.count }
84 
+
85 
+      @event.payload['foo']['bar']['baz'] = "a2b"
86 
+      lambda {
87 
+        @checker.receive([@event])
88 
+      }.should change { Event.count }.by(1)
89 
+
90 
+      @event.payload['foo']['bar']['baz'] = "a b"
91 
+      lambda {
92 
+        @checker.receive([@event])
93 
+      }.should change { Event.count }.by(1)
94 
+    end
95 
+
74 96 
     it "handles negated regex" do
75 97 
       @event.payload['foo']['bar']['baz'] = "a2b"
76 98 
       @checker.options['rules'][0] = {
 
@@ -89,6 +111,24 @@ describe Agents::TriggerAgent do
89 111 
       }.should change { Event.count }.by(1)
90 112 
     end
91 113
114 
+    it "handles array of negated regex" do
115 
+      @event.payload['foo']['bar']['baz'] = "a2b"
116 
+      @checker.options['rules'][0] = {
117 
+        'type' => "!regex",
118 
+        'value' => ["a\\db", "a2b"],
119 
+        'path' => "foo.bar.baz",
120 
+      }
121 
+
122 
+      lambda {
123 
+        @checker.receive([@event])
124 
+      }.should_not change { Event.count }
125 
+
126 
+      @event.payload['foo']['bar']['baz'] = "a3b"
127 
+      lambda {
128 
+        @checker.receive([@event])
129 
+      }.should change { Event.count }.by(1)
130 
+    end
131 
+
92 132 
     it "puts can extract values into the message based on paths" do
93 133 
       @checker.receive([@event])
94 134 
       Event.last.payload['message'].should == "I saw 'a2b' from Joe"
 
@@ -109,6 +149,21 @@ describe Agents::TriggerAgent do
109 149 
       }.should_not change { Event.count }
110 150 
     end
111 151
152 
+    it "handles array of numerical comparisons" do
153 
+      @event.payload['foo']['bar']['baz'] = "5"
154 
+      @checker.options['rules'].first['value'] = [6, 3]
155 
+      @checker.options['rules'].first['type'] = "field<value"
156 
+
157 
+      lambda {
158 
+        @checker.receive([@event])
159 
+      }.should change { Event.count }.by(1)
160 
+
161 
+      @checker.options['rules'].first['value'] = [4, 3]
162 
+      lambda {
163 
+        @checker.receive([@event])
164 
+      }.should_not change { Event.count }
165 
+    end
166 
+
112 167 
     it "handles exact comparisons" do
113 168 
       @event.payload['foo']['bar']['baz'] = "hello world"
114 169 
       @checker.options['rules'].first['type'] = "field==value"
 
@@ -124,6 +179,21 @@ describe Agents::TriggerAgent do
124 179 
       }.should change { Event.count }.by(1)
125 180 
     end
126 181
182 
+    it "handles array of exact comparisons" do
183 
+      @event.payload['foo']['bar']['baz'] = "hello world"
184 
+      @checker.options['rules'].first['type'] = "field==value"
185 
+
186 
+      @checker.options['rules'].first['value'] = ["hello there", "hello universe"]
187 
+      lambda {
188 
+        @checker.receive([@event])
189 
+      }.should_not change { Event.count }
190 
+
191 
+      @checker.options['rules'].first['value'] = ["hello world", "hello universe"]
192 
+      lambda {
193 
+        @checker.receive([@event])
194 
+      }.should change { Event.count }.by(1)
195 
+    end
196 
+
127 197 
     it "handles negated comparisons" do
128 198 
       @event.payload['foo']['bar']['baz'] = "hello world"
129 199 
       @checker.options['rules'].first['type'] = "field!=value"
 
@@ -140,6 +210,22 @@ describe Agents::TriggerAgent do
140 210 
       }.should change { Event.count }.by(1)
141 211 
     end
142 212
213 
+    it "handles array of negated comparisons" do
214 
+      @event.payload['foo']['bar']['baz'] = "hello world"
215 
+      @checker.options['rules'].first['type'] = "field!=value"
216 
+      @checker.options['rules'].first['value'] = ["hello world", "hello world"]
217 
+
218 
+      lambda {
219 
+        @checker.receive([@event])
220 
+      }.should_not change { Event.count }
221 
+
222 
+      @checker.options['rules'].first['value'] = ["hello there", "hello world"]
223 
+
224 
+      lambda {
225 
+        @checker.receive([@event])
226 
+      }.should change { Event.count }.by(1)
227 
+    end
228 
+
143 229 
     it "does fine without dots in the path" do
144 230 
       @event.payload = { 'hello' => "world" }
145 231 
       @checker.options['rules'].first['type'] = "field==value"

+ 47 - 1
spec/models/agents/website_agent_spec.rb
View File 

@@ -331,11 +331,26 @@ describe Agents::WebsiteAgent do
331 331 
         end
332 332 
       end
333 333 
     end
334 
+
335 
+    describe "#receive" do
336 
+      it "should scrape from the url element in incoming event payload" do
337 
+        @event = Event.new
338 
+        @event.agent = agents(:bob_rain_notifier_agent)
339 
+        @event.payload = { 'url' => "http://xkcd.com" }
340 
+
341 
+        lambda {
342 
+          @checker.options = @site
343 
+          @checker.receive([@event])
344 
+        }.should change { Event.count }.by(1)
345 
+      end
346 
+    end
334 347 
   end
335 348
336 349 
   describe "checking with http basic auth" do
337 350 
     before do
338 
-      stub_request(:any, /user:pass/).to_return(:body => File.read(Rails.root.join("spec/data_fixtures/xkcd.html")), :status => 200)
351 
+      stub_request(:any, /example/).
352 
+        with(headers: { 'Authorization' => "Basic #{['user:pass'].pack('m').chomp}" }).
353 
+        to_return(:body => File.read(Rails.root.join("spec/data_fixtures/xkcd.html")), :status => 200)
339 354 
       @site = {
340 355 
         'name' => "XKCD",
341 356 
         'expected_update_period_in_days' => 2,
 
@@ -361,4 +376,35 @@ describe Agents::WebsiteAgent do
361 376 
       end
362 377 
     end
363 378 
   end
379 
+
380 
+  describe "checking with User-Agent" do
381 
+    before do
382 
+      stub_request(:any, /example/).
383 
+        with(headers: { 'User-Agent' => 'Sushi' }).
384 
+        to_return(:body => File.read(Rails.root.join("spec/data_fixtures/xkcd.html")), :status => 200)
385 
+      @site = {
386 
+        'name' => "XKCD",
387 
+        'expected_update_period_in_days' => 2,
388 
+        'type' => "html",
389 
+        'url' => "http://www.example.com",
390 
+        'mode' => 'on_change',
391 
+        'extract' => {
392 
+          'url' => { 'css' => "#comic img", 'attr' => "src" },
393 
+          'title' => { 'css' => "#comic img", 'attr' => "alt" },
394 
+          'hovertext' => { 'css' => "#comic img", 'attr' => "title" }
395 
+        },
396 
+        'user_agent' => "Sushi"
397 
+      }
398 
+      @checker = Agents::WebsiteAgent.new(:name => "ua", :options => @site)
399 
+      @checker.user = users(:bob)
400 
+      @checker.save!
401 
+    end
402 
+
403 
+    describe "#check" do
404 
+      it "should check for changes" do
405 
+        lambda { @checker.check }.should change { Event.count }.by(1)
406 
+        lambda { @checker.check }.should_not change { Event.count }
407 
+      end
408 
+    end
409 
+  end
364 410 
 end