Merge branch 'master' of https://github.com/cantino/huginn
* 'master' of https://github.com/cantino/huginn: (154 commits)
Skip exclude_replies validation
do not delete running jobs
remove spring loads too
integrate @knu's suggestions
Supress pop up emails
Update twitter_user_agent_spec.rb
fix exclude_replies?
There is a button to add a new Agent directly in a Scenario; deleting an Agent from its show page no longer 404s; there is a button to destroy all delayed_jobs
remove spring because it frequently conflicts with foreman
local email was not configured correctly without this change
Make exclude_replies configurable
Intercept email in development by default
Improve language for clarity
Test operates correctly regardless of dev's ENV config
Provide a default value in the case that the ENV variable isn't set
Adds the ability to allow public signup without invite code
Revert "Make a `spring` group which user must opt in to"
Revert "Replace the `platform :ruby18` hack with `install_if`"
Replace the `platform :ruby18` hack with `install_if`
Make a `spring` group which user must opt in to
...
# Conflicts:
# app/models/agents/data_output_agent.rb
# app/models/agents/rss_agent.rb
Dustin Miller
111 changed files with 2681 additions and 576 deletions
+ 14
- 4
.env.example
@@ -47,6 +47,9 @@ FORCE_SSL=false |
||
| 47 | 47 |
# You can see its use in user.rb. PLEASE CHANGE THIS! |
| 48 | 48 |
INVITATION_CODE=try-huginn |
| 49 | 49 |
|
| 50 |
+# If you don't want to require new users to have an invitation code in order to sign up, set this to true. |
|
| 51 |
+SKIP_INVITATION_CODE=false |
|
| 52 |
+ |
|
| 50 | 53 |
############################# |
| 51 | 54 |
# Email Configuration # |
| 52 | 55 |
############################# |
@@ -54,7 +57,7 @@ INVITATION_CODE=try-huginn |
||
| 54 | 57 |
# Outgoing email settings. To use Gmail or Google Apps, put your Google Apps domain or gmail.com |
| 55 | 58 |
# as the SMTP_DOMAIN and your Gmail username and password as the SMTP_USER_NAME and SMTP_PASSWORD. |
| 56 | 59 |
# |
| 57 |
-# PLEASE NOTE: In order to enable emails locally (e.g., when not in the production Rails environment), |
|
| 60 |
+# PLEASE NOTE: In order to enable sending real emails via SMTP locally (e.g., when not in the production Rails environment), |
|
| 58 | 61 |
# you must also set SEND_EMAIL_IN_DEVELOPMENT to true below. |
| 59 | 62 |
# |
| 60 | 63 |
# If you have trouble with port 587 on Gmail, you can also try setting |
@@ -68,7 +71,8 @@ SMTP_PORT=587 |
||
| 68 | 71 |
SMTP_AUTHENTICATION=plain |
| 69 | 72 |
SMTP_ENABLE_STARTTLS_AUTO=true |
| 70 | 73 |
|
| 71 |
-# Send emails when running in the development Rails environment. |
|
| 74 |
+# Set to true to send real emails via SMTP when running in the development Rails environment. |
|
| 75 |
+# Set to false to have emails intercepted in development and displayed at http://localhost:3000/letter_opener |
|
| 72 | 76 |
SEND_EMAIL_IN_DEVELOPMENT=false |
| 73 | 77 |
|
| 74 | 78 |
# The address from which system emails will appear to be sent. |
@@ -88,6 +92,8 @@ AGENT_LOG_LENGTH=200 |
||
| 88 | 92 |
|
| 89 | 93 |
TWITTER_OAUTH_KEY= |
| 90 | 94 |
TWITTER_OAUTH_SECRET= |
| 95 |
+TWITTER_CONSUMER_KEY= |
|
| 96 |
+TWITTER_CONSUMER_SECRET= |
|
| 91 | 97 |
|
| 92 | 98 |
THIRTY_SEVEN_SIGNALS_OAUTH_KEY= |
| 93 | 99 |
THIRTY_SEVEN_SIGNALS_OAUTH_SECRET= |
@@ -149,6 +155,10 @@ ENABLE_SECOND_PRECISION_SCHEDULE=false |
||
| 149 | 155 |
# at the expense of time accuracy. |
| 150 | 156 |
SCHEDULER_FREQUENCY=0.3 |
| 151 | 157 |
|
| 158 |
+# Specify the frequency with which the scheduler checks for and cleans up expired events. |
|
| 159 |
+# You can use `m` for minutes, `h` for hours, and `d` for days. |
|
| 160 |
+EVENT_EXPIRATION_CHECK=6h |
|
| 161 |
+ |
|
| 152 | 162 |
# Use Graphviz for generating diagrams instead of using Google Chart |
| 153 | 163 |
# Tools. Specify a dot(1) command path built with SVG support |
| 154 | 164 |
# enabled. |
@@ -161,7 +171,7 @@ TIMEZONE="Pacific Time (US & Canada)" |
||
| 161 | 171 |
FAILED_JOBS_TO_KEEP=100 |
| 162 | 172 |
|
| 163 | 173 |
# Maximum runtime of background jobs in minutes |
| 164 |
-DELAYED_JOB_MAX_RUNTIME=20 |
|
| 174 |
+DELAYED_JOB_MAX_RUNTIME=2 |
|
| 165 | 175 |
|
| 166 | 176 |
# Amount of seconds for delayed_job to sleep before checking for new jobs |
| 167 |
-DELAYED_JOB_SLEEP_DELAY=10 |
|
| 177 |
+DELAYED_JOB_SLEEP_DELAY=10 |
+ 38
- 0
CHANGES.md
@@ -1,5 +1,43 @@ |
||
| 1 | 1 |
# Changes |
| 2 | 2 |
|
| 3 |
+* Jul 30, 2015 - RssAgent can configure the order of events created via `events_order`. |
|
| 4 |
+* Jul 29, 2015 - WebsiteAgent can configure the order of events created via `events_order`. |
|
| 5 |
+* Jul 29, 2015 - DataOutputAgent can configure the order of events in the output via `events_order`. |
|
| 6 |
+* Jul 20, 2015 - Control Links (used by the SchedularAgent) are correctly exported in Scenarios. |
|
| 7 |
+* Jul 20, 2015 - keep\_events\_for was moved from days to seconds; Scenarios have a schema verison. |
|
| 8 |
+* Jul 8, 2015 - DataOutputAgent supports feed icon, and a new template variable `events`. |
|
| 9 |
+* Jul 1, 2015 - DeDuplicationAgent properly handles destruction of memory. |
|
| 10 |
+* Jun 26, 2015 - Add `max_events_per_run` to RssAgent. |
|
| 11 |
+* Jun 19, 2015 - Add `url_from_event` to WebsiteAgent. |
|
| 12 |
+* Jun 17, 2015 - RssAgent emits events for new feed items in chronological order. |
|
| 13 |
+* Jun 17, 2015 - Liquid filter `unescape` added. |
|
| 14 |
+* Jun 17, 2015 - Liquid filter `regex_replace` and `regex_replace_first` added, with escape sequence support. |
|
| 15 |
+* Jun 15, 2015 - Liquid filter `uri_expand` added. |
|
| 16 |
+* Jun 13, 2015 - Liquid templating engine is upgraded to version 3. |
|
| 17 |
+* Jun 12, 2015 - RSSAgent can now accept an array of URLs. |
|
| 18 |
+* Jun 8, 2015 - WebsiteAgent includes a `use_namespaces` option to enable XML namespaces. |
|
| 19 |
+* May 27, 2015 - Validation warns user if they have not provided a `path` when using JSONPath in WebsiteAgent. |
|
| 20 |
+* May 24, 2015 - Show Agents' name and user in the jobs panel. |
|
| 21 |
+* May 19, 2015 - Add "Dry Run" to the action menu. |
|
| 22 |
+* May 23, 2015 - JavaScriptAgent has dry run and inline syntax highlighting JavaScript and CoffeeScript. |
|
| 23 |
+* May 11, 2015 - Make delayed\_job sleep\_delay and max\_run\_time .env configurable. |
|
| 24 |
+* May 9, 2015 - Add 'unescapeHTML' functionality to the javascript agent. |
|
| 25 |
+* May 3, 2015 - Use ActiveJobs interface. |
|
| 26 |
+* Apr 28, 2015 - Adds Wunderlist agent. |
|
| 27 |
+* Apr 25, 2015 - Allow user to clear memory of an agent. |
|
| 28 |
+* Apr 25, 2015 - Allow WebsiteAgent to unzip compressed JSON. |
|
| 29 |
+* Apr 12, 2015 - Allow the webhook agent to loop over returned results if the payload\_path points to an array. |
|
| 30 |
+* Mar 27, 2015 - Add wit.ai Agent. |
|
| 31 |
+* Mar 24, 2015 - CloudFoundry integration. |
|
| 32 |
+* Mar 20, 2015 - Upgrade to Rails 4.2. |
|
| 33 |
+* Mar 17, 2015 - Add new "Dry Run" feature for some Agents. |
|
| 34 |
+* Feb 26, 2015 - Update to PushBullet API version 2. |
|
| 35 |
+* Feb 22, 2015 - Allow Agents to request immediate propagation of Events. |
|
| 36 |
+* Feb 18, 2015 - Convert \n to actual line breaks after interpolating liquid and add `line_break_tag`. |
|
| 37 |
+* Feb 6, 2015 - Allow UserLocationAgent to accept `min_distance` to require a certain distance traveled. |
|
| 38 |
+* Feb 1, 2015 - Allow a `body` key to be provided to set email body in the EmailAgent. |
|
| 39 |
+* Jan 21, 2015 - Allow custom icon for Slack webhooks. |
|
| 40 |
+* Jan 20, 2015 - Add `max_accuracy` to UserLocationAgent. |
|
| 3 | 41 |
* Jan 19, 2015 - WebRequestConcern Agents can supply `disable_ssl_verification` to disable ssl verification. |
| 4 | 42 |
* Jan 13, 2015 - Docker image updated. |
| 5 | 43 |
* Jan 8, 2015 - Allow toggling of accuracy when displaying locations in the UserLocationAgent map. |
+ 12
- 9
Gemfile
@@ -1,5 +1,8 @@ |
||
| 1 | 1 |
source 'https://rubygems.org' |
| 2 | 2 |
|
| 3 |
+# Ruby 2.0 is the minimum requirement |
|
| 4 |
+ruby ['2.0.0', RUBY_VERSION].max |
|
| 5 |
+ |
|
| 3 | 6 |
# Optional libraries. To conserve RAM, comment out any that you don't need, |
| 4 | 7 |
# then run `bundle` and commit the updated Gemfile and Gemfile.lock. |
| 5 | 8 |
gem 'twilio-ruby', '~> 3.11.5' # TwilioAgent |
@@ -63,26 +66,27 @@ gem 'em-http-request', '~> 1.1.2' |
||
| 63 | 66 |
gem 'faraday', '~> 0.9.0' |
| 64 | 67 |
gem 'faraday_middleware' |
| 65 | 68 |
gem 'feed-normalizer' |
| 66 |
-gem 'font-awesome-sass', '~> 4.3' |
|
| 69 |
+gem 'font-awesome-sass', '~> 4.3.2' |
|
| 67 | 70 |
gem 'foreman', '~> 0.63.0' |
| 68 | 71 |
# geokit-rails doesn't work with geokit 1.8.X but it specifies ~> 1.5 |
| 69 | 72 |
# in its own Gemfile. |
| 70 | 73 |
gem 'geokit', '~> 1.8.4' |
| 71 | 74 |
gem 'geokit-rails', '~> 2.0.1' |
| 72 | 75 |
gem 'httparty', '~> 0.13' |
| 73 |
-gem 'jquery-rails', '~> 3.1.0' |
|
| 76 |
+gem 'jquery-rails', '~> 3.1.3' |
|
| 74 | 77 |
gem 'json', '~> 1.8.1' |
| 75 | 78 |
gem 'jsonpath', '~> 0.5.6' |
| 76 | 79 |
gem 'kaminari', '~> 0.16.1' |
| 77 | 80 |
gem 'kramdown', '~> 1.3.3' |
| 78 |
-gem 'liquid', '~> 2.6.1' |
|
| 81 |
+gem 'liquid', '~> 3.0.3' |
|
| 82 |
+gem 'mini_magick' |
|
| 79 | 83 |
gem 'mysql2', '~> 0.3.16' |
| 80 | 84 |
gem 'multi_xml' |
| 81 | 85 |
gem 'nokogiri', '~> 1.6.4' |
| 82 | 86 |
gem 'omniauth' |
| 83 |
-gem 'rails' , '4.2.1' |
|
| 87 |
+gem 'rails' , '4.2.2' |
|
| 84 | 88 |
gem 'rufus-scheduler', '~> 3.0.8', require: false |
| 85 |
-gem 'sass-rails', '~> 5.0' |
|
| 89 |
+gem 'sass-rails', '~> 5.0.3' |
|
| 86 | 90 |
gem 'select2-rails', '~> 3.5.4' |
| 87 | 91 |
gem 'spectrum-rails' |
| 88 | 92 |
gem 'string-scrub' # for ruby <2.1 |
@@ -97,26 +101,25 @@ group :development do |
||
| 97 | 101 |
gem 'guard' |
| 98 | 102 |
gem 'guard-livereload' |
| 99 | 103 |
gem 'guard-rspec' |
| 104 |
+ gem 'letter_opener_web' |
|
| 100 | 105 |
|
| 101 | 106 |
group :test do |
| 102 | 107 |
gem 'coveralls', require: false |
| 103 | 108 |
gem 'delorean' |
| 104 |
- gem 'pry' |
|
| 109 |
+ gem 'pry-rails' |
|
| 105 | 110 |
gem 'rr' |
| 106 | 111 |
gem 'rspec', '~> 3.2' |
| 107 | 112 |
gem 'rspec-collection_matchers', '~> 1.1.0' |
| 108 | 113 |
gem 'rspec-rails', '~> 3.1' |
| 109 | 114 |
gem 'rspec-html-matchers', '~> 0.7' |
| 110 | 115 |
gem 'shoulda-matchers' |
| 111 |
- gem 'spring', '~> 1.3.0' |
|
| 112 |
- gem 'spring-commands-rspec' |
|
| 113 | 116 |
gem 'vcr' |
| 114 | 117 |
gem 'webmock', '~> 1.17.4', require: false |
| 115 | 118 |
end |
| 116 | 119 |
end |
| 117 | 120 |
|
| 118 | 121 |
group :production do |
| 119 |
- gem 'rack' |
|
| 122 |
+ gem 'rack', '> 1.5.0' |
|
| 120 | 123 |
end |
| 121 | 124 |
|
| 122 | 125 |
# Platform requirements. |
+ 65
- 60
Gemfile.lock
@@ -32,36 +32,36 @@ GEM |
||
| 32 | 32 |
remote: https://rubygems.org/ |
| 33 | 33 |
specs: |
| 34 | 34 |
ace-rails-ap (2.0.1) |
| 35 |
- actionmailer (4.2.1) |
|
| 36 |
- actionpack (= 4.2.1) |
|
| 37 |
- actionview (= 4.2.1) |
|
| 38 |
- activejob (= 4.2.1) |
|
| 35 |
+ actionmailer (4.2.2) |
|
| 36 |
+ actionpack (= 4.2.2) |
|
| 37 |
+ actionview (= 4.2.2) |
|
| 38 |
+ activejob (= 4.2.2) |
|
| 39 | 39 |
mail (~> 2.5, >= 2.5.4) |
| 40 | 40 |
rails-dom-testing (~> 1.0, >= 1.0.5) |
| 41 |
- actionpack (4.2.1) |
|
| 42 |
- actionview (= 4.2.1) |
|
| 43 |
- activesupport (= 4.2.1) |
|
| 41 |
+ actionpack (4.2.2) |
|
| 42 |
+ actionview (= 4.2.2) |
|
| 43 |
+ activesupport (= 4.2.2) |
|
| 44 | 44 |
rack (~> 1.6) |
| 45 | 45 |
rack-test (~> 0.6.2) |
| 46 | 46 |
rails-dom-testing (~> 1.0, >= 1.0.5) |
| 47 | 47 |
rails-html-sanitizer (~> 1.0, >= 1.0.1) |
| 48 |
- actionview (4.2.1) |
|
| 49 |
- activesupport (= 4.2.1) |
|
| 48 |
+ actionview (4.2.2) |
|
| 49 |
+ activesupport (= 4.2.2) |
|
| 50 | 50 |
builder (~> 3.1) |
| 51 | 51 |
erubis (~> 2.7.0) |
| 52 | 52 |
rails-dom-testing (~> 1.0, >= 1.0.5) |
| 53 | 53 |
rails-html-sanitizer (~> 1.0, >= 1.0.1) |
| 54 |
- activejob (4.2.1) |
|
| 55 |
- activesupport (= 4.2.1) |
|
| 54 |
+ activejob (4.2.2) |
|
| 55 |
+ activesupport (= 4.2.2) |
|
| 56 | 56 |
globalid (>= 0.3.0) |
| 57 |
- activemodel (4.2.1) |
|
| 58 |
- activesupport (= 4.2.1) |
|
| 57 |
+ activemodel (4.2.2) |
|
| 58 |
+ activesupport (= 4.2.2) |
|
| 59 | 59 |
builder (~> 3.1) |
| 60 |
- activerecord (4.2.1) |
|
| 61 |
- activemodel (= 4.2.1) |
|
| 62 |
- activesupport (= 4.2.1) |
|
| 60 |
+ activerecord (4.2.2) |
|
| 61 |
+ activemodel (= 4.2.2) |
|
| 62 |
+ activesupport (= 4.2.2) |
|
| 63 | 63 |
arel (~> 6.0) |
| 64 |
- activesupport (4.2.1) |
|
| 64 |
+ activesupport (4.2.2) |
|
| 65 | 65 |
i18n (~> 0.7) |
| 66 | 66 |
json (~> 1.7, >= 1.7.7) |
| 67 | 67 |
minitest (~> 5.1) |
@@ -159,7 +159,7 @@ GEM |
||
| 159 | 159 |
hpricot (>= 0.6) |
| 160 | 160 |
simple-rss (>= 1.1) |
| 161 | 161 |
ffi (1.9.5) |
| 162 |
- font-awesome-sass (4.3.1) |
|
| 162 |
+ font-awesome-sass (4.3.2.1) |
|
| 163 | 163 |
sass (~> 3.2) |
| 164 | 164 |
forecast_io (2.0.0) |
| 165 | 165 |
faraday |
@@ -174,7 +174,7 @@ GEM |
||
| 174 | 174 |
geokit-rails (2.0.1) |
| 175 | 175 |
geokit (~> 1.5) |
| 176 | 176 |
rails (>= 3.0) |
| 177 |
- globalid (0.3.3) |
|
| 177 |
+ globalid (0.3.5) |
|
| 178 | 178 |
activesupport (>= 4.1.0) |
| 179 | 179 |
google-api-client (0.7.1) |
| 180 | 180 |
addressable (>= 2.3.2) |
@@ -202,7 +202,6 @@ GEM |
||
| 202 | 202 |
rspec (>= 2.14, < 4.0) |
| 203 | 203 |
hashie (2.0.5) |
| 204 | 204 |
haversine (0.3.0) |
| 205 |
- hike (1.2.3) |
|
| 206 | 205 |
hipchat (1.2.0) |
| 207 | 206 |
httparty |
| 208 | 207 |
hitimes (1.2.2) |
@@ -221,11 +220,11 @@ GEM |
||
| 221 | 220 |
hypdf (1.0.7) |
| 222 | 221 |
httmultiparty (= 0.3.10) |
| 223 | 222 |
i18n (0.7.0) |
| 224 |
- jquery-rails (3.1.1) |
|
| 223 |
+ jquery-rails (3.1.3) |
|
| 225 | 224 |
railties (>= 3.0, < 5.0) |
| 226 | 225 |
thor (>= 0.14, < 2.0) |
| 227 |
- json (1.8.2) |
|
| 228 |
- jsonpath (0.5.6) |
|
| 226 |
+ json (1.8.3) |
|
| 227 |
+ jsonpath (0.5.7) |
|
| 229 | 228 |
multi_json |
| 230 | 229 |
jwt (1.4.1) |
| 231 | 230 |
kaminari (0.16.1) |
@@ -235,13 +234,19 @@ GEM |
||
| 235 | 234 |
kramdown (1.3.3) |
| 236 | 235 |
launchy (2.4.2) |
| 237 | 236 |
addressable (~> 2.3) |
| 237 |
+ letter_opener (1.4.1) |
|
| 238 |
+ launchy (~> 2.2) |
|
| 239 |
+ letter_opener_web (1.3.0) |
|
| 240 |
+ actionmailer (>= 3.2) |
|
| 241 |
+ letter_opener (~> 1.0) |
|
| 242 |
+ railties (>= 3.2) |
|
| 238 | 243 |
libv8 (3.16.14.7) |
| 239 |
- liquid (2.6.2) |
|
| 244 |
+ liquid (3.0.6) |
|
| 240 | 245 |
listen (2.7.9) |
| 241 | 246 |
celluloid (>= 0.15.2) |
| 242 | 247 |
rb-fsevent (>= 0.9.3) |
| 243 | 248 |
rb-inotify (>= 0.9) |
| 244 |
- loofah (2.0.1) |
|
| 249 |
+ loofah (2.0.2) |
|
| 245 | 250 |
nokogiri (>= 1.5.9) |
| 246 | 251 |
lumberjack (1.0.9) |
| 247 | 252 |
macaddr (1.7.1) |
@@ -251,11 +256,12 @@ GEM |
||
| 251 | 256 |
memoizable (0.4.2) |
| 252 | 257 |
thread_safe (~> 0.3, >= 0.3.1) |
| 253 | 258 |
method_source (0.8.2) |
| 254 |
- mime-types (2.5) |
|
| 259 |
+ mime-types (2.6.1) |
|
| 260 |
+ mini_magick (4.2.3) |
|
| 255 | 261 |
mini_portile (0.6.2) |
| 256 |
- minitest (5.5.1) |
|
| 262 |
+ minitest (5.7.0) |
|
| 257 | 263 |
mqtt (0.3.1) |
| 258 |
- multi_json (1.11.0) |
|
| 264 |
+ multi_json (1.11.2) |
|
| 259 | 265 |
multi_xml (0.5.5) |
| 260 | 266 |
multipart-post (2.0.0) |
| 261 | 267 |
mysql2 (0.3.16) |
@@ -301,21 +307,23 @@ GEM |
||
| 301 | 307 |
coderay (~> 1.1.0) |
| 302 | 308 |
method_source (~> 0.8.1) |
| 303 | 309 |
slop (~> 3.4) |
| 310 |
+ pry-rails (0.3.4) |
|
| 311 |
+ pry (>= 0.9.10) |
|
| 304 | 312 |
quiet_assets (1.1.0) |
| 305 | 313 |
railties (>= 3.1, < 5.0) |
| 306 |
- rack (1.6.1) |
|
| 314 |
+ rack (1.6.4) |
|
| 307 | 315 |
rack-test (0.6.3) |
| 308 | 316 |
rack (>= 1.0) |
| 309 |
- rails (4.2.1) |
|
| 310 |
- actionmailer (= 4.2.1) |
|
| 311 |
- actionpack (= 4.2.1) |
|
| 312 |
- actionview (= 4.2.1) |
|
| 313 |
- activejob (= 4.2.1) |
|
| 314 |
- activemodel (= 4.2.1) |
|
| 315 |
- activerecord (= 4.2.1) |
|
| 316 |
- activesupport (= 4.2.1) |
|
| 317 |
+ rails (4.2.2) |
|
| 318 |
+ actionmailer (= 4.2.2) |
|
| 319 |
+ actionpack (= 4.2.2) |
|
| 320 |
+ actionview (= 4.2.2) |
|
| 321 |
+ activejob (= 4.2.2) |
|
| 322 |
+ activemodel (= 4.2.2) |
|
| 323 |
+ activerecord (= 4.2.2) |
|
| 324 |
+ activesupport (= 4.2.2) |
|
| 317 | 325 |
bundler (>= 1.3.0, < 2.0) |
| 318 |
- railties (= 4.2.1) |
|
| 326 |
+ railties (= 4.2.2) |
|
| 319 | 327 |
sprockets-rails |
| 320 | 328 |
rails-deprecated_sanitizer (1.0.3) |
| 321 | 329 |
activesupport (>= 4.2.0.alpha) |
@@ -330,9 +338,9 @@ GEM |
||
| 330 | 338 |
rails_stdout_logging |
| 331 | 339 |
rails_serve_static_assets (0.0.4) |
| 332 | 340 |
rails_stdout_logging (0.0.3) |
| 333 |
- railties (4.2.1) |
|
| 334 |
- actionpack (= 4.2.1) |
|
| 335 |
- activesupport (= 4.2.1) |
|
| 341 |
+ railties (4.2.2) |
|
| 342 |
+ actionpack (= 4.2.2) |
|
| 343 |
+ activesupport (= 4.2.2) |
|
| 336 | 344 |
rake (>= 0.8.7) |
| 337 | 345 |
thor (>= 0.18.1, < 2.0) |
| 338 | 346 |
raindrops (0.13.0) |
@@ -384,8 +392,8 @@ GEM |
||
| 384 | 392 |
rufus-scheduler (3.0.9) |
| 385 | 393 |
tzinfo |
| 386 | 394 |
safe_yaml (1.0.4) |
| 387 |
- sass (3.4.12) |
|
| 388 |
- sass-rails (5.0.1) |
|
| 395 |
+ sass (3.4.14) |
|
| 396 |
+ sass-rails (5.0.3) |
|
| 389 | 397 |
railties (>= 4.0.0, < 5.0) |
| 390 | 398 |
sass (~> 3.1) |
| 391 | 399 |
sprockets (>= 2.8, < 4.0) |
@@ -411,15 +419,9 @@ GEM |
||
| 411 | 419 |
slop (3.6.0) |
| 412 | 420 |
spectrum-rails (1.3.4) |
| 413 | 421 |
railties (>= 3.1) |
| 414 |
- spring (1.3.6) |
|
| 415 |
- spring-commands-rspec (1.0.4) |
|
| 416 |
- spring (>= 0.9.1) |
|
| 417 |
- sprockets (2.12.3) |
|
| 418 |
- hike (~> 1.2) |
|
| 419 |
- multi_json (~> 1.0) |
|
| 422 |
+ sprockets (3.2.0) |
|
| 420 | 423 |
rack (~> 1.0) |
| 421 |
- tilt (~> 1.1, != 1.3.0) |
|
| 422 |
- sprockets-rails (2.2.4) |
|
| 424 |
+ sprockets-rails (2.3.1) |
|
| 423 | 425 |
actionpack (>= 3.0) |
| 424 | 426 |
activesupport (>= 3.0) |
| 425 | 427 |
sprockets (>= 2.8, < 4.0) |
@@ -512,7 +514,7 @@ DEPENDENCIES |
||
| 512 | 514 |
faraday_middleware |
| 513 | 515 |
feed-normalizer |
| 514 | 516 |
ffi (>= 1.9.4) |
| 515 |
- font-awesome-sass (~> 4.3) |
|
| 517 |
+ font-awesome-sass (~> 4.3.2) |
|
| 516 | 518 |
forecast_io (~> 2.0.0) |
| 517 | 519 |
foreman (~> 0.63.0) |
| 518 | 520 |
geokit (~> 1.8.4) |
@@ -525,12 +527,14 @@ DEPENDENCIES |
||
| 525 | 527 |
hipchat (~> 1.2.0) |
| 526 | 528 |
httparty (~> 0.13) |
| 527 | 529 |
hypdf (~> 1.0.7) |
| 528 |
- jquery-rails (~> 3.1.0) |
|
| 530 |
+ jquery-rails (~> 3.1.3) |
|
| 529 | 531 |
json (~> 1.8.1) |
| 530 | 532 |
jsonpath (~> 0.5.6) |
| 531 | 533 |
kaminari (~> 0.16.1) |
| 532 | 534 |
kramdown (~> 1.3.3) |
| 533 |
- liquid (~> 2.6.1) |
|
| 535 |
+ letter_opener_web |
|
| 536 |
+ liquid (~> 3.0.3) |
|
| 537 |
+ mini_magick |
|
| 534 | 538 |
mqtt |
| 535 | 539 |
multi_xml |
| 536 | 540 |
mysql2 (~> 0.3.16) |
@@ -544,10 +548,10 @@ DEPENDENCIES |
||
| 544 | 548 |
omniauth-wunderlist! |
| 545 | 549 |
pg |
| 546 | 550 |
protected_attributes (~> 1.0.8) |
| 547 |
- pry |
|
| 551 |
+ pry-rails |
|
| 548 | 552 |
quiet_assets |
| 549 |
- rack |
|
| 550 |
- rails (= 4.2.1) |
|
| 553 |
+ rack (> 1.5.0) |
|
| 554 |
+ rails (= 4.2.2) |
|
| 551 | 555 |
rails_12factor |
| 552 | 556 |
rr |
| 553 | 557 |
rspec (~> 3.2) |
@@ -557,13 +561,11 @@ DEPENDENCIES |
||
| 557 | 561 |
rturk (~> 2.12.1) |
| 558 | 562 |
ruby-growl (~> 4.1.0) |
| 559 | 563 |
rufus-scheduler (~> 3.0.8) |
| 560 |
- sass-rails (~> 5.0) |
|
| 564 |
+ sass-rails (~> 5.0.3) |
|
| 561 | 565 |
select2-rails (~> 3.5.4) |
| 562 | 566 |
shoulda-matchers |
| 563 | 567 |
slack-notifier (~> 1.0.0) |
| 564 | 568 |
spectrum-rails |
| 565 |
- spring (~> 1.3.0) |
|
| 566 |
- spring-commands-rspec |
|
| 567 | 569 |
string-scrub |
| 568 | 570 |
therubyracer (~> 0.12.2) |
| 569 | 571 |
tumblr_client |
@@ -580,3 +582,6 @@ DEPENDENCIES |
||
| 580 | 582 |
weibo_2! |
| 581 | 583 |
wunderground (~> 1.2.0) |
| 582 | 584 |
xmpp4r (~> 0.5.6) |
| 585 |
+ |
|
| 586 |
+BUNDLED WITH |
|
| 587 |
+ 1.10.6 |
+ 1
- 1
Guardfile
@@ -8,7 +8,7 @@ guard 'livereload' do |
||
| 8 | 8 |
watch(%r{(app|vendor)(/assets/\w+/(.+\.(css|js|html|png|jpg))).*}) { |m| "/assets/#{m[3]}" }
|
| 9 | 9 |
end |
| 10 | 10 |
|
| 11 |
-guard :rspec, cmd: 'bundle exec spring rspec' do |
|
| 11 |
+guard :rspec, cmd: 'bundle exec rspec' do |
|
| 12 | 12 |
watch(%r{^spec/.+_spec\.rb$})
|
| 13 | 13 |
watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
|
| 14 | 14 |
watch('spec/spec_helper.rb') { "spec" }
|
+ 3
- 3
Procfile
@@ -1,5 +1,5 @@ |
||
| 1 | 1 |
# Procfile for development using the new threaded worker (scheduler, twitter stream and delayed job) |
| 2 |
-web: bundle exec rails server |
|
| 2 |
+web: bundle exec rails server -b0.0.0.0 |
|
| 3 | 3 |
jobs: bundle exec rails runner bin/threaded.rb |
| 4 | 4 |
|
| 5 | 5 |
# Possible Profile configuration for production: |
@@ -7,7 +7,7 @@ jobs: bundle exec rails runner bin/threaded.rb |
||
| 7 | 7 |
# jobs: bundle exec rails runner bin/threaded.rb |
| 8 | 8 |
|
| 9 | 9 |
# Old version with separate processes (use this if you have issues with the threaded version) |
| 10 |
-# web: bundle exec rails server |
|
| 10 |
+# web: bundle exec rails server -b0.0.0.0 |
|
| 11 | 11 |
# schedule: bundle exec rails runner bin/schedule.rb |
| 12 | 12 |
# twitter: bundle exec rails runner bin/twitter_stream.rb |
| 13 |
-# dj: bundle exec script/delayed_job run |
|
| 13 |
+# dj: bundle exec script/delayed_job run |
+ 7
- 3
README.md
@@ -66,7 +66,9 @@ If you just want to play around, you can simply fork this repository, then perfo |
||
| 66 | 66 |
* Read the [wiki][wiki] for usage examples and to get started making new Agents. |
| 67 | 67 |
* Periodically run `git fetch upstream` and then `git checkout master && git merge upstream/master` to merge in the newest version of Huginn. |
| 68 | 68 |
|
| 69 |
-Note: by default, emails are not sent in the `development` Rails environment, which is what you just setup. If you'd like to enable emails when playing with Huginn locally, set `SEND_EMAIL_IN_DEVELOPMENT` to `true` in your `.env` file. |
|
| 69 |
+Note: By default, emails are intercepted in the `development` Rails environment, which is what you just setup. You can view |
|
| 70 |
+them at [http://localhost:3000/letter_opener](http://localhost:3000/letter_opener). If you'd like to send real emails via SMTP when playing |
|
| 71 |
+with Huginn locally, set `SEND_EMAIL_IN_DEVELOPMENT` to `true` in your `.env` file. |
|
| 70 | 72 |
|
| 71 | 73 |
If you need more detailed instructions, see the [Novice setup guide][novice-setup-guide]. |
| 72 | 74 |
|
@@ -80,9 +82,11 @@ All agents have specs! Test all specs with `bundle exec rspec`, or test a specif |
||
| 80 | 82 |
|
| 81 | 83 |
## Deployment |
| 82 | 84 |
|
| 83 |
-[](https://heroku.com/deploy) (Takes a few minutes to setup. Be sure to click 'View it' after launch!) |
|
| 85 |
+Try Huginn on Heroku: [](https://heroku.com/deploy) (Takes a few minutes to setup. Be sure to click 'View it' after launch!) |
|
| 84 | 86 |
|
| 85 |
-Huginn can run on Heroku for free! Please see [the Huginn Wiki](https://github.com/cantino/huginn/wiki#deploying-huginn) for detailed deployment strategies for different providers. |
|
| 87 |
+Huginn works on the free version of Heroku [with limitations](https://github.com/cantino/huginn/wiki/Run-Huginn-for-free-on-Heroku). For non-experimental use, we recommend Heroku's cheapest paid plan or our Docker container. |
|
| 88 |
+ |
|
| 89 |
+Please see [the Huginn Wiki](https://github.com/cantino/huginn/wiki#deploying-huginn) for detailed deployment strategies for different providers. |
|
| 86 | 90 |
|
| 87 | 91 |
### Optional Setup |
| 88 | 92 |
|
+ 1
- 0
app.json
@@ -3,6 +3,7 @@ |
||
| 3 | 3 |
"description": "Build agents that monitor and act on your behalf. Your agents are standing by!", |
| 4 | 4 |
"website": "https://github.com/cantino/huginn", |
| 5 | 5 |
"repository": "https://github.com/cantino/huginn", |
| 6 |
+ "logo": "https://raw.githubusercontent.com/cantino/huginn/master/media/huginn-icon-64.png", |
|
| 6 | 7 |
"env": {
|
| 7 | 8 |
"BUILDPACK_URL": "https://github.com/heroku/heroku-buildpack-multi.git", |
| 8 | 9 |
"APP_SECRET_TOKEN": {
|
+ 61
- 4
app/assets/javascripts/components/utils.js.coffee
@@ -34,10 +34,67 @@ class @Utils |
||
| 34 | 34 |
body?(modal.querySelector('.modal-body'))
|
| 35 | 35 |
$(modal).modal('show')
|
| 36 | 36 |
|
| 37 |
- @handleDryRunButton: (button, data = $(button.form).serialize()) -> |
|
| 37 |
+ @handleDryRunButton: (button, data = if button.form then $(':input[name!="_method"]', button.form).serialize() else '') ->
|
|
| 38 | 38 |
$(button).prop('disabled', true)
|
| 39 |
+ cleanup = -> $(button).prop('disabled', false)
|
|
| 40 |
+ |
|
| 41 |
+ url = $(button).data('action-url')
|
|
| 42 |
+ with_event_mode = $(button).data('with-event-mode')
|
|
| 43 |
+ |
|
| 44 |
+ if with_event_mode is 'no' |
|
| 45 |
+ return @invokeDryRun(url, data, cleanup) |
|
| 46 |
+ |
|
| 47 |
+ Utils.showDynamicModal """ |
|
| 48 |
+ <h5>Event to send#{if with_event_mode is 'maybe' then ' (Optional)' else ''}</h5>
|
|
| 49 |
+ <form class="dry-run-form" method="post"> |
|
| 50 |
+ <div class="form-group"> |
|
| 51 |
+ <textarea rows="10" name="event" class="payload-editor" data-height="200"> |
|
| 52 |
+ {}
|
|
| 53 |
+ </textarea> |
|
| 54 |
+ </div> |
|
| 55 |
+ <div class="form-group"> |
|
| 56 |
+ <input value="Dry Run" class="btn btn-primary" type="submit" /> |
|
| 57 |
+ </div> |
|
| 58 |
+ </form> |
|
| 59 |
+ """, |
|
| 60 |
+ body: (body) => |
|
| 61 |
+ form = $(body).find('.dry-run-form')
|
|
| 62 |
+ payload_editor = form.find('.payload-editor')
|
|
| 63 |
+ if previous = $(button).data('payload')
|
|
| 64 |
+ payload_editor.text(previous) |
|
| 65 |
+ window.setupJsonEditor(payload_editor) |
|
| 66 |
+ form.submit (e) => |
|
| 67 |
+ e.preventDefault() |
|
| 68 |
+ json = $(e.target).find('.payload-editor').val()
|
|
| 69 |
+ json = '{}' if json == ''
|
|
| 70 |
+ try |
|
| 71 |
+ payload = JSON.parse(json) |
|
| 72 |
+ throw true unless payload.constructor is Object |
|
| 73 |
+ if Object.keys(payload).length == 0 |
|
| 74 |
+ json = '' |
|
| 75 |
+ else |
|
| 76 |
+ json = JSON.stringify(payload) |
|
| 77 |
+ catch |
|
| 78 |
+ alert 'Invalid JSON object.' |
|
| 79 |
+ return |
|
| 80 |
+ if json == '' |
|
| 81 |
+ if with_event_mode is 'yes' |
|
| 82 |
+ alert 'Event is required for this agent to run.' |
|
| 83 |
+ return |
|
| 84 |
+ dry_run_data = data |
|
| 85 |
+ $(button).data('payload', null)
|
|
| 86 |
+ else |
|
| 87 |
+ dry_run_data = "event=#{encodeURIComponent(json)}&#{data}"
|
|
| 88 |
+ $(button).data('payload', json)
|
|
| 89 |
+ $(body).closest('[role=dialog]').on 'hidden.bs.modal', =>
|
|
| 90 |
+ @invokeDryRun(url, dry_run_data, cleanup) |
|
| 91 |
+ .modal('hide')
|
|
| 92 |
+ title: 'Dry Run' |
|
| 93 |
+ onHide: cleanup |
|
| 94 |
+ |
|
| 95 |
+ @invokeDryRun: (url, data, callback) -> |
|
| 39 | 96 |
$('body').css(cursor: 'progress')
|
| 40 |
- $.ajax type: 'POST', url: $(button).data('action-url'), dataType: 'json', data: data
|
|
| 97 |
+ $.ajax type: 'POST', url: url, dataType: 'json', data: data |
|
| 41 | 98 |
.always => |
| 42 | 99 |
$('body').css(cursor: 'auto')
|
| 43 | 100 |
.done (json) => |
@@ -55,7 +112,7 @@ class @Utils |
||
| 55 | 112 |
find('.agent-dry-run-events').text(json.events).end().
|
| 56 | 113 |
find('.agent-dry-run-memory').text(json.memory)
|
| 57 | 114 |
title: 'Dry Run Results', |
| 58 |
- onHide: -> $(button).prop('disabled', false)
|
|
| 115 |
+ onHide: callback |
|
| 59 | 116 |
.fail (xhr, status, error) -> |
| 60 | 117 |
alert('Error: ' + error)
|
| 61 |
- $(button).prop('disabled', false)
|
|
| 118 |
+ callback() |
+ 6
- 0
app/assets/stylesheets/application.css.scss.erb
@@ -27,6 +27,12 @@ body { padding-top: 60px; }
|
||
| 27 | 27 |
margin-bottom: 100px; |
| 28 | 28 |
} |
| 29 | 29 |
|
| 30 |
+.well.description {
|
|
| 31 |
+ h1 { font-size: 30px; }
|
|
| 32 |
+ h2 { font-size: 26px; }
|
|
| 33 |
+ h3 { font-size: 22px; }
|
|
| 34 |
+} |
|
| 35 |
+ |
|
| 30 | 36 |
/* Rails scaffold style compatibility */ |
| 31 | 37 |
#error_explanation {
|
| 32 | 38 |
color: #f00; |
+ 31
- 15
app/concerns/dry_runnable.rb
@@ -1,10 +1,8 @@ |
||
| 1 | 1 |
module DryRunnable |
| 2 |
- def dry_run! |
|
| 3 |
- readonly! |
|
| 2 |
+ extend ActiveSupport::Concern |
|
| 4 | 3 |
|
| 5 |
- class << self |
|
| 6 |
- prepend Sandbox |
|
| 7 |
- end |
|
| 4 |
+ def dry_run!(event = nil) |
|
| 5 |
+ @dry_run = true |
|
| 8 | 6 |
|
| 9 | 7 |
log = StringIO.new |
| 10 | 8 |
@dry_run_logger = Logger.new(log) |
@@ -14,7 +12,13 @@ module DryRunnable |
||
| 14 | 12 |
|
| 15 | 13 |
begin |
| 16 | 14 |
raise "#{short_type} does not support dry-run" unless can_dry_run?
|
| 17 |
- check |
|
| 15 |
+ readonly! |
|
| 16 |
+ if event |
|
| 17 |
+ raise "This agent cannot receive an event!" unless can_receive_events? |
|
| 18 |
+ receive([event]) |
|
| 19 |
+ else |
|
| 20 |
+ check |
|
| 21 |
+ end |
|
| 18 | 22 |
rescue => e |
| 19 | 23 |
error "Exception during dry-run. #{e.message}: #{e.backtrace.join("\n")}"
|
| 20 | 24 |
end |
@@ -23,28 +27,38 @@ module DryRunnable |
||
| 23 | 27 |
memory: memory, |
| 24 | 28 |
log: log.string, |
| 25 | 29 |
) |
| 30 |
+ ensure |
|
| 31 |
+ @dry_run = false |
|
| 26 | 32 |
end |
| 27 | 33 |
|
| 28 | 34 |
def dry_run? |
| 29 |
- is_a? Sandbox |
|
| 35 |
+ !!@dry_run |
|
| 36 |
+ end |
|
| 37 |
+ |
|
| 38 |
+ included do |
|
| 39 |
+ prepend Wrapper |
|
| 30 | 40 |
end |
| 31 | 41 |
|
| 32 |
- module Sandbox |
|
| 42 |
+ module Wrapper |
|
| 33 | 43 |
attr_accessor :results |
| 34 | 44 |
|
| 35 | 45 |
def logger |
| 46 |
+ return super unless dry_run? |
|
| 36 | 47 |
@dry_run_logger |
| 37 | 48 |
end |
| 38 | 49 |
|
| 39 |
- def save |
|
| 40 |
- valid? |
|
| 50 |
+ def save(options = {})
|
|
| 51 |
+ return super unless dry_run? |
|
| 52 |
+ perform_validations(options) |
|
| 41 | 53 |
end |
| 42 | 54 |
|
| 43 |
- def save! |
|
| 44 |
- save or raise ActiveRecord::RecordNotSaved |
|
| 55 |
+ def save!(options = {})
|
|
| 56 |
+ return super unless dry_run? |
|
| 57 |
+ save(options) or raise_record_invalid |
|
| 45 | 58 |
end |
| 46 | 59 |
|
| 47 | 60 |
def log(message, options = {})
|
| 61 |
+ return super unless dry_run? |
|
| 48 | 62 |
case options[:level] || 3 |
| 49 | 63 |
when 0..2 |
| 50 | 64 |
sev = Logger::DEBUG |
@@ -57,10 +71,12 @@ module DryRunnable |
||
| 57 | 71 |
logger.log(sev, message) |
| 58 | 72 |
end |
| 59 | 73 |
|
| 60 |
- def create_event(event_hash) |
|
| 74 |
+ def create_event(event) |
|
| 75 |
+ return super unless dry_run? |
|
| 61 | 76 |
if can_create_events? |
| 62 |
- @dry_run_results[:events] << event_hash[:payload] |
|
| 63 |
- events.build({ user: user, expires_at: new_event_expiration_date }.merge(event_hash))
|
|
| 77 |
+ event = build_event(event) |
|
| 78 |
+ @dry_run_results[:events] << event.payload |
|
| 79 |
+ event |
|
| 64 | 80 |
else |
| 65 | 81 |
error "This Agent cannot create events!" |
| 66 | 82 |
end |
+ 139
- 1
app/concerns/liquid_interpolatable.rb
@@ -15,7 +15,9 @@ module LiquidInterpolatable |
||
| 15 | 15 |
interpolated |
| 16 | 16 |
rescue Liquid::Error => e |
| 17 | 17 |
errors.add(:options, "has an error with Liquid templating: #{e.message}")
|
| 18 |
- false |
|
| 18 |
+ rescue |
|
| 19 |
+ # Calling `interpolated` without an incoming may naturally fail |
|
| 20 |
+ # with various errors when an agent expects one. |
|
| 19 | 21 |
end |
| 20 | 22 |
|
| 21 | 23 |
# Return the current interpolation context. Use this in your Agent |
@@ -132,6 +134,65 @@ module LiquidInterpolatable |
||
| 132 | 134 |
nil |
| 133 | 135 |
end |
| 134 | 136 |
|
| 137 |
+ # Get the destination URL of a given URL by recursively following |
|
| 138 |
+ # redirects, up to 5 times in a row. If a given string is not a |
|
| 139 |
+ # valid absolute HTTP URL or in case of too many redirects, the |
|
| 140 |
+ # original string is returned. If any network/protocol error |
|
| 141 |
+ # occurs while following redirects, the last URL followed is |
|
| 142 |
+ # returned. |
|
| 143 |
+ def uri_expand(url, limit = 5) |
|
| 144 |
+ case url |
|
| 145 |
+ when URI |
|
| 146 |
+ uri = url |
|
| 147 |
+ else |
|
| 148 |
+ url = url.to_s |
|
| 149 |
+ begin |
|
| 150 |
+ uri = URI(url) |
|
| 151 |
+ rescue URI::Error |
|
| 152 |
+ return url |
|
| 153 |
+ end |
|
| 154 |
+ end |
|
| 155 |
+ |
|
| 156 |
+ http = Faraday.new do |builder| |
|
| 157 |
+ builder.adapter :net_http |
|
| 158 |
+ # builder.use FaradayMiddleware::FollowRedirects, limit: limit |
|
| 159 |
+ # ...does not handle non-HTTP URLs. |
|
| 160 |
+ end |
|
| 161 |
+ |
|
| 162 |
+ limit.times do |
|
| 163 |
+ begin |
|
| 164 |
+ case uri |
|
| 165 |
+ when URI::HTTP |
|
| 166 |
+ return uri.to_s unless uri.host |
|
| 167 |
+ response = http.head(uri) |
|
| 168 |
+ case response.status |
|
| 169 |
+ when 301, 302, 303, 307 |
|
| 170 |
+ if location = response['location'] |
|
| 171 |
+ uri += location |
|
| 172 |
+ next |
|
| 173 |
+ end |
|
| 174 |
+ end |
|
| 175 |
+ end |
|
| 176 |
+ rescue URI::Error, Faraday::Error, SystemCallError => e |
|
| 177 |
+ logger.error "#{e.class} in #{__method__}(#{url.inspect}) [uri=#{uri.to_s.inspect}]: #{e.message}:\n#{e.backtrace.join("\n")}"
|
|
| 178 |
+ end |
|
| 179 |
+ |
|
| 180 |
+ return uri.to_s |
|
| 181 |
+ end |
|
| 182 |
+ |
|
| 183 |
+ logger.error "Too many rediretions in #{__method__}(#{url.inspect}) [uri=#{uri.to_s.inspect}]"
|
|
| 184 |
+ |
|
| 185 |
+ url |
|
| 186 |
+ end |
|
| 187 |
+ |
|
| 188 |
+ # Unescape (basic) HTML entities in a string |
|
| 189 |
+ # |
|
| 190 |
+ # This currently decodes the following entities only: "'", |
|
| 191 |
+ # """, "<", ">", "&", "&#dd;" and "&#xhh;". |
|
| 192 |
+ def unescape(input) |
|
| 193 |
+ CGI.unescapeHTML(input) rescue input |
|
| 194 |
+ end |
|
| 195 |
+ |
|
| 135 | 196 |
# Escape a string for use in XPath expression |
| 136 | 197 |
def to_xpath(string) |
| 137 | 198 |
subs = string.to_s.scan(/\G(?:\A\z|[^"]+|[^']+)/).map { |x|
|
@@ -148,6 +209,83 @@ module LiquidInterpolatable |
||
| 148 | 209 |
'concat(' << subs.join(', ') << ')'
|
| 149 | 210 |
end |
| 150 | 211 |
end |
| 212 |
+ |
|
| 213 |
+ def regex_replace(input, regex, replacement = nil) |
|
| 214 |
+ input.to_s.gsub(Regexp.new(regex), unescape_replacement(replacement.to_s)) |
|
| 215 |
+ end |
|
| 216 |
+ |
|
| 217 |
+ def regex_replace_first(input, regex, replacement = nil) |
|
| 218 |
+ input.to_s.sub(Regexp.new(regex), unescape_replacement(replacement.to_s)) |
|
| 219 |
+ end |
|
| 220 |
+ |
|
| 221 |
+ private |
|
| 222 |
+ |
|
| 223 |
+ def logger |
|
| 224 |
+ @@logger ||= |
|
| 225 |
+ if defined?(Rails) |
|
| 226 |
+ Rails.logger |
|
| 227 |
+ else |
|
| 228 |
+ require 'logger' |
|
| 229 |
+ Logger.new(STDERR) |
|
| 230 |
+ end |
|
| 231 |
+ end |
|
| 232 |
+ |
|
| 233 |
+ BACKSLASH = "\\".freeze |
|
| 234 |
+ |
|
| 235 |
+ UNESCAPE = {
|
|
| 236 |
+ "a" => "\a", |
|
| 237 |
+ "b" => "\b", |
|
| 238 |
+ "e" => "\e", |
|
| 239 |
+ "f" => "\f", |
|
| 240 |
+ "n" => "\n", |
|
| 241 |
+ "r" => "\r", |
|
| 242 |
+ "s" => " ", |
|
| 243 |
+ "t" => "\t", |
|
| 244 |
+ "v" => "\v", |
|
| 245 |
+ } |
|
| 246 |
+ |
|
| 247 |
+ # Unescape a replacement text for use in the second argument of |
|
| 248 |
+ # gsub/sub. The following escape sequences are recognized: |
|
| 249 |
+ # |
|
| 250 |
+ # - "\\" (backslash itself) |
|
| 251 |
+ # - "\a" (alert) |
|
| 252 |
+ # - "\b" (backspace) |
|
| 253 |
+ # - "\e" (escape) |
|
| 254 |
+ # - "\f" (form feed) |
|
| 255 |
+ # - "\n" (new line) |
|
| 256 |
+ # - "\r" (carriage return) |
|
| 257 |
+ # - "\s" (space) |
|
| 258 |
+ # - "\t" (horizontal tab) |
|
| 259 |
+ # - "\u{XXXX}" (unicode codepoint)
|
|
| 260 |
+ # - "\v" (vertical tab) |
|
| 261 |
+ # - "\xXX" (hexadecimal character) |
|
| 262 |
+ # - "\1".."\9" (numbered capture groups) |
|
| 263 |
+ # - "\+" (last capture group) |
|
| 264 |
+ # - "\k<name>" (named capture group) |
|
| 265 |
+ # - "\&" or "\0" (complete matched text) |
|
| 266 |
+ # - "\`" (string before match) |
|
| 267 |
+ # - "\'" (string after match) |
|
| 268 |
+ # |
|
| 269 |
+ # Octal escape sequences are deliberately unsupported to avoid |
|
| 270 |
+ # conflict with numbered capture groups. Rather obscure Emacs |
|
| 271 |
+ # style character codes ("\C-x", "\M-\C-x" etc.) are also omitted
|
|
| 272 |
+ # from this implementation. |
|
| 273 |
+ def unescape_replacement(s) |
|
| 274 |
+ s.gsub(/\\(?:([\d+&`'\\]|k<\w+>)|u\{([[:xdigit:]]+)\}|x([[:xdigit:]]{2})|(.))/) {
|
|
| 275 |
+ if c = $1 |
|
| 276 |
+ BACKSLASH + c |
|
| 277 |
+ elsif c = ($2 && [$2.to_i(16)].pack('U')) ||
|
|
| 278 |
+ ($3 && [$3.to_i(16)].pack('C'))
|
|
| 279 |
+ if c == BACKSLASH |
|
| 280 |
+ BACKSLASH + c |
|
| 281 |
+ else |
|
| 282 |
+ c |
|
| 283 |
+ end |
|
| 284 |
+ else |
|
| 285 |
+ UNESCAPE[$4] || $4 |
|
| 286 |
+ end |
|
| 287 |
+ } |
|
| 288 |
+ end |
|
| 151 | 289 |
end |
| 152 | 290 |
Liquid::Template.register_filter(LiquidInterpolatable::Filters) |
| 153 | 291 |
|
+ 161
- 0
app/concerns/sortable_events.rb
@@ -0,0 +1,161 @@ |
||
| 1 |
+module SortableEvents |
|
| 2 |
+ extend ActiveSupport::Concern |
|
| 3 |
+ |
|
| 4 |
+ included do |
|
| 5 |
+ validate :validate_events_order |
|
| 6 |
+ end |
|
| 7 |
+ |
|
| 8 |
+ def description_events_order(*args) |
|
| 9 |
+ self.class.description_events_order(*args) |
|
| 10 |
+ end |
|
| 11 |
+ |
|
| 12 |
+ module ClassMethods |
|
| 13 |
+ def can_order_created_events! |
|
| 14 |
+ raise if cannot_create_events? |
|
| 15 |
+ prepend AutomaticSorter |
|
| 16 |
+ end |
|
| 17 |
+ |
|
| 18 |
+ def can_order_created_events? |
|
| 19 |
+ include? AutomaticSorter |
|
| 20 |
+ end |
|
| 21 |
+ |
|
| 22 |
+ def cannot_order_created_events? |
|
| 23 |
+ !can_order_created_events? |
|
| 24 |
+ end |
|
| 25 |
+ |
|
| 26 |
+ def description_events_order(events = 'events created in each run') |
|
| 27 |
+ <<-MD.lstrip |
|
| 28 |
+ To specify the order of #{events}, set `events_order` to an array of sort keys, each of which looks like either `expression` or `[expression, type, descending]`, as described as follows:
|
|
| 29 |
+ |
|
| 30 |
+ * _expression_ is a Liquid template to generate a string to be used as sort key. |
|
| 31 |
+ |
|
| 32 |
+ * _type_ (optional) is one of `string` (default), `number` and `time`, which specifies how to evaluate _expression_ for comparison. |
|
| 33 |
+ |
|
| 34 |
+ * _descending_ (optional) is a boolean value to determine if comparison should be done in descending (reverse) order, which defaults to `false`. |
|
| 35 |
+ |
|
| 36 |
+ Sort keys listed earlier take precedence over ones listed later. For example, if you want to sort articles by the date and then by the author, specify `[["{{date}}", "time"], "{{author}}"]`.
|
|
| 37 |
+ |
|
| 38 |
+ Sorting is done stably, so even if all events have the same set of sort key values the original order is retained. Also, a special Liquid variable `_index_` is provided, which contains the zero-based index number of each event, which means you can exactly reverse the order of events by specifying `[["{{_index_}}", "number", true]]`.
|
|
| 39 |
+ MD |
|
| 40 |
+ end |
|
| 41 |
+ end |
|
| 42 |
+ |
|
| 43 |
+ def can_order_created_events? |
|
| 44 |
+ self.class.can_order_created_events? |
|
| 45 |
+ end |
|
| 46 |
+ |
|
| 47 |
+ def cannot_order_created_events? |
|
| 48 |
+ self.class.cannot_order_created_events? |
|
| 49 |
+ end |
|
| 50 |
+ |
|
| 51 |
+ def events_order |
|
| 52 |
+ options['events_order'] |
|
| 53 |
+ end |
|
| 54 |
+ |
|
| 55 |
+ module AutomaticSorter |
|
| 56 |
+ def check |
|
| 57 |
+ return super unless events_order |
|
| 58 |
+ sorting_events do |
|
| 59 |
+ super |
|
| 60 |
+ end |
|
| 61 |
+ end |
|
| 62 |
+ |
|
| 63 |
+ def receive(incoming_events) |
|
| 64 |
+ return super unless events_order |
|
| 65 |
+ # incoming events should be processed sequentially |
|
| 66 |
+ incoming_events.each do |event| |
|
| 67 |
+ sorting_events do |
|
| 68 |
+ super([event]) |
|
| 69 |
+ end |
|
| 70 |
+ end |
|
| 71 |
+ end |
|
| 72 |
+ |
|
| 73 |
+ def create_event(event) |
|
| 74 |
+ if @sortable_events |
|
| 75 |
+ event = build_event(event) |
|
| 76 |
+ @sortable_events << event |
|
| 77 |
+ event |
|
| 78 |
+ else |
|
| 79 |
+ super |
|
| 80 |
+ end |
|
| 81 |
+ end |
|
| 82 |
+ |
|
| 83 |
+ private |
|
| 84 |
+ |
|
| 85 |
+ def sorting_events(&block) |
|
| 86 |
+ @sortable_events = [] |
|
| 87 |
+ yield |
|
| 88 |
+ ensure |
|
| 89 |
+ events, @sortable_events = @sortable_events, nil |
|
| 90 |
+ sort_events(events).each do |event| |
|
| 91 |
+ create_event(event) |
|
| 92 |
+ end |
|
| 93 |
+ end |
|
| 94 |
+ end |
|
| 95 |
+ |
|
| 96 |
+ private |
|
| 97 |
+ |
|
| 98 |
+ EXPRESSION_PARSER = {
|
|
| 99 |
+ 'string' => ->string { string },
|
|
| 100 |
+ 'number' => ->string { string.to_f },
|
|
| 101 |
+ 'time' => ->string { Time.zone.parse(string) },
|
|
| 102 |
+ } |
|
| 103 |
+ EXPRESSION_TYPES = EXPRESSION_PARSER.keys.freeze |
|
| 104 |
+ |
|
| 105 |
+ def validate_events_order |
|
| 106 |
+ case order_by = events_order |
|
| 107 |
+ when nil |
|
| 108 |
+ when Array |
|
| 109 |
+ # Each tuple may be either [expression, type, desc] or just |
|
| 110 |
+ # expression. |
|
| 111 |
+ order_by.each do |expression, type, desc| |
|
| 112 |
+ case expression |
|
| 113 |
+ when String |
|
| 114 |
+ # ok |
|
| 115 |
+ else |
|
| 116 |
+ errors.add(:base, "first element of each events_order tuple must be a Liquid template") |
|
| 117 |
+ break |
|
| 118 |
+ end |
|
| 119 |
+ case type |
|
| 120 |
+ when nil, *EXPRESSION_TYPES |
|
| 121 |
+ # ok |
|
| 122 |
+ else |
|
| 123 |
+ errors.add(:base, "second element of each events_order tuple must be #{EXPRESSION_TYPES.to_sentence(last_word_connector: ' or ')}")
|
|
| 124 |
+ break |
|
| 125 |
+ end |
|
| 126 |
+ if !desc.nil? && boolify(desc).nil? |
|
| 127 |
+ errors.add(:base, "third element of each events_order tuple must be a boolean value") |
|
| 128 |
+ break |
|
| 129 |
+ end |
|
| 130 |
+ end |
|
| 131 |
+ else |
|
| 132 |
+ errors.add(:base, "events_order must be an array of arrays") |
|
| 133 |
+ end |
|
| 134 |
+ end |
|
| 135 |
+ |
|
| 136 |
+ # Sort given events in order specified by the "events_order" option |
|
| 137 |
+ def sort_events(events) |
|
| 138 |
+ order_by = events_order.presence or |
|
| 139 |
+ return events |
|
| 140 |
+ |
|
| 141 |
+ orders = order_by.map { |_, _, desc = false| boolify(desc) }
|
|
| 142 |
+ |
|
| 143 |
+ Utils.sort_tuples!( |
|
| 144 |
+ events.map.with_index { |event, index|
|
|
| 145 |
+ interpolate_with(event) {
|
|
| 146 |
+ interpolation_context['_index_'] = index |
|
| 147 |
+ order_by.map { |expression, type, _|
|
|
| 148 |
+ string = interpolate_string(expression) |
|
| 149 |
+ begin |
|
| 150 |
+ EXPRESSION_PARSER[type || 'string'.freeze][string] |
|
| 151 |
+ rescue |
|
| 152 |
+ error "Cannot parse #{string.inspect} as #{type}; treating it as string"
|
|
| 153 |
+ string |
|
| 154 |
+ end |
|
| 155 |
+ } |
|
| 156 |
+ } << index << event # index is to make sorting stable |
|
| 157 |
+ }, |
|
| 158 |
+ orders |
|
| 159 |
+ ).collect!(&:last) |
|
| 160 |
+ end |
|
| 161 |
+end |
+ 79
- 0
app/concerns/web_request_concern.rb
@@ -2,6 +2,58 @@ require 'faraday' |
||
| 2 | 2 |
require 'faraday_middleware' |
| 3 | 3 |
|
| 4 | 4 |
module WebRequestConcern |
| 5 |
+ module DoNotEncoder |
|
| 6 |
+ def self.encode(params) |
|
| 7 |
+ params.map do |key, value| |
|
| 8 |
+ value.nil? ? "#{key}" : "#{key}=#{value}"
|
|
| 9 |
+ end.join('&')
|
|
| 10 |
+ end |
|
| 11 |
+ |
|
| 12 |
+ def self.decode(val) |
|
| 13 |
+ [val] |
|
| 14 |
+ end |
|
| 15 |
+ end |
|
| 16 |
+ |
|
| 17 |
+ class CharacterEncoding < Faraday::Middleware |
|
| 18 |
+ def initialize(app, force_encoding: nil, default_encoding: nil, unzip: nil) |
|
| 19 |
+ super(app) |
|
| 20 |
+ @force_encoding = force_encoding |
|
| 21 |
+ @default_encoding = default_encoding |
|
| 22 |
+ @unzip = unzip |
|
| 23 |
+ end |
|
| 24 |
+ |
|
| 25 |
+ def call(env) |
|
| 26 |
+ @app.call(env).on_complete do |env| |
|
| 27 |
+ body = env[:body] |
|
| 28 |
+ |
|
| 29 |
+ case @unzip |
|
| 30 |
+ when 'gzip'.freeze |
|
| 31 |
+ body.replace(ActiveSupport::Gzip.decompress(body)) |
|
| 32 |
+ end |
|
| 33 |
+ |
|
| 34 |
+ case |
|
| 35 |
+ when @force_encoding |
|
| 36 |
+ encoding = @force_encoding |
|
| 37 |
+ when body.encoding == Encoding::ASCII_8BIT |
|
| 38 |
+ # Not all Faraday adapters support automatic charset |
|
| 39 |
+ # detection, so we do that. |
|
| 40 |
+ case env[:response_headers][:content_type] |
|
| 41 |
+ when /;\s*charset\s*=\s*([^()<>@,;:\\\"\/\[\]?={}\s]+)/i
|
|
| 42 |
+ encoding = Encoding.find($1) rescue nil |
|
| 43 |
+ when /\A\s*(?:text\/[^\s;]+|application\/(?:[^\s;]+\+)?(?:xml|json))\s*(?:;|\z)/i |
|
| 44 |
+ encoding = @default_encoding |
|
| 45 |
+ else |
|
| 46 |
+ # Never try to transcode a binary content |
|
| 47 |
+ return |
|
| 48 |
+ end |
|
| 49 |
+ end |
|
| 50 |
+ body.encode!(Encoding::UTF_8, encoding) unless body.encoding == Encoding::UTF_8 |
|
| 51 |
+ end |
|
| 52 |
+ end |
|
| 53 |
+ end |
|
| 54 |
+ |
|
| 55 |
+ Faraday::Response.register_middleware character_encoding: CharacterEncoding |
|
| 56 |
+ |
|
| 5 | 57 |
extend ActiveSupport::Concern |
| 6 | 58 |
|
| 7 | 59 |
def validate_web_request_options! |
@@ -22,6 +74,23 @@ module WebRequestConcern |
||
| 22 | 74 |
rescue ArgumentError => e |
| 23 | 75 |
errors.add(:base, e.message) |
| 24 | 76 |
end |
| 77 |
+ |
|
| 78 |
+ if (encoding = options['force_encoding']).present? |
|
| 79 |
+ case encoding |
|
| 80 |
+ when String |
|
| 81 |
+ begin |
|
| 82 |
+ Encoding.find(encoding) |
|
| 83 |
+ rescue ArgumentError |
|
| 84 |
+ errors.add(:base, "Unknown encoding: #{encoding.inspect}")
|
|
| 85 |
+ end |
|
| 86 |
+ else |
|
| 87 |
+ errors.add(:base, "force_encoding must be a string") |
|
| 88 |
+ end |
|
| 89 |
+ end |
|
| 90 |
+ end |
|
| 91 |
+ |
|
| 92 |
+ def default_encoding |
|
| 93 |
+ Encoding::UTF_8 |
|
| 25 | 94 |
end |
| 26 | 95 |
|
| 27 | 96 |
def faraday |
@@ -32,12 +101,22 @@ module WebRequestConcern |
||
| 32 | 101 |
} |
| 33 | 102 |
|
| 34 | 103 |
@faraday ||= Faraday.new(faraday_options) { |builder|
|
| 104 |
+ builder.response :character_encoding, |
|
| 105 |
+ force_encoding: interpolated['force_encoding'].presence, |
|
| 106 |
+ default_encoding: default_encoding, |
|
| 107 |
+ unzip: interpolated['unzip'].presence |
|
| 108 |
+ |
|
| 35 | 109 |
builder.headers = headers if headers.length > 0 |
| 36 | 110 |
|
| 37 | 111 |
builder.headers[:user_agent] = user_agent |
| 38 | 112 |
|
| 39 | 113 |
builder.use FaradayMiddleware::FollowRedirects |
| 40 | 114 |
builder.request :url_encoded |
| 115 |
+ |
|
| 116 |
+ if boolify(interpolated['disable_url_encoding']) |
|
| 117 |
+ builder.options.params_encoder = DoNotEncoder |
|
| 118 |
+ end |
|
| 119 |
+ |
|
| 41 | 120 |
if userinfo = basic_auth_credentials |
| 42 | 121 |
builder.request :basic_auth, *userinfo |
| 43 | 122 |
end |
+ 28
- 12
app/controllers/agents_controller.rb
@@ -37,7 +37,7 @@ class AgentsController < ApplicationController |
||
| 37 | 37 |
def dry_run |
| 38 | 38 |
attrs = params[:agent] || {}
|
| 39 | 39 |
if agent = current_user.agents.find_by(id: params[:id]) |
| 40 |
- # PUT /agents/:id/dry_run |
|
| 40 |
+ # POST /agents/:id/dry_run |
|
| 41 | 41 |
if attrs.present? |
| 42 | 42 |
type = agent.type |
| 43 | 43 |
agent = Agent.build_for_type(type, current_user, attrs) |
@@ -50,7 +50,13 @@ class AgentsController < ApplicationController |
||
| 50 | 50 |
agent.name ||= '(Untitled)' |
| 51 | 51 |
|
| 52 | 52 |
if agent.valid? |
| 53 |
- results = agent.dry_run! |
|
| 53 |
+ if event_payload = params[:event] |
|
| 54 |
+ dummy_agent = Agent.build_for_type('ManualEventAgent', current_user, name: 'Dry-Runner')
|
|
| 55 |
+ dummy_agent.readonly! |
|
| 56 |
+ event = dummy_agent.events.build(user: current_user, payload: event_payload) |
|
| 57 |
+ end |
|
| 58 |
+ |
|
| 59 |
+ results = agent.dry_run!(event) |
|
| 54 | 60 |
|
| 55 | 61 |
render json: {
|
| 56 | 62 |
log: results[:log], |
@@ -142,6 +148,9 @@ class AgentsController < ApplicationController |
||
| 142 | 148 |
else |
| 143 | 149 |
@agent = agents.build |
| 144 | 150 |
end |
| 151 |
+ |
|
| 152 |
+ @agent.scenario_ids = [params[:scenario_id]] if params[:scenario_id] && current_user.scenarios.find_by(id: params[:scenario_id]) |
|
| 153 |
+ |
|
| 145 | 154 |
initialize_presenter |
| 146 | 155 |
|
| 147 | 156 |
respond_to do |format| |
@@ -160,7 +169,7 @@ class AgentsController < ApplicationController |
||
| 160 | 169 |
|
| 161 | 170 |
respond_to do |format| |
| 162 | 171 |
if @agent.save |
| 163 |
- format.html { redirect_back "'#{@agent.name}' was successfully created." }
|
|
| 172 |
+ format.html { redirect_back "'#{@agent.name}' was successfully created.", return: agents_path }
|
|
| 164 | 173 |
format.json { render json: @agent, status: :ok, location: agent_path(@agent) }
|
| 165 | 174 |
else |
| 166 | 175 |
initialize_presenter |
@@ -175,7 +184,7 @@ class AgentsController < ApplicationController |
||
| 175 | 184 |
|
| 176 | 185 |
respond_to do |format| |
| 177 | 186 |
if @agent.update_attributes(params[:agent]) |
| 178 |
- format.html { redirect_back "'#{@agent.name}' was successfully updated." }
|
|
| 187 |
+ format.html { redirect_back "'#{@agent.name}' was successfully updated.", return: agents_path }
|
|
| 179 | 188 |
format.json { render json: @agent, status: :ok, location: agent_path(@agent) }
|
| 180 | 189 |
else |
| 181 | 190 |
initialize_presenter |
@@ -225,16 +234,23 @@ class AgentsController < ApplicationController |
||
| 225 | 234 |
protected |
| 226 | 235 |
|
| 227 | 236 |
# Sanitize params[:return] to prevent open redirect attacks, a common security issue. |
| 228 |
- def redirect_back(message) |
|
| 229 |
- if params[:return] == "show" && @agent && !@agent.destroyed? |
|
| 230 |
- path = agent_path(@agent) |
|
| 231 |
- elsif params[:return] =~ /\A#{Regexp::escape scenarios_path}\/\d+\Z/
|
|
| 232 |
- path = params[:return] |
|
| 233 |
- else |
|
| 234 |
- path = agents_path |
|
| 237 |
+ def redirect_back(message, options = {})
|
|
| 238 |
+ case ret = params[:return] || options[:return] |
|
| 239 |
+ when "show" |
|
| 240 |
+ if @agent && !@agent.destroyed? |
|
| 241 |
+ path = agent_path(@agent) |
|
| 242 |
+ else |
|
| 243 |
+ path = agents_path |
|
| 244 |
+ end |
|
| 245 |
+ when /\A#{Regexp::escape scenarios_path}\/\d+\z/, agents_path
|
|
| 246 |
+ path = ret |
|
| 235 | 247 |
end |
| 236 | 248 |
|
| 237 |
- redirect_to path, notice: message |
|
| 249 |
+ if path |
|
| 250 |
+ redirect_to path, notice: message |
|
| 251 |
+ else |
|
| 252 |
+ super agents_path, notice: message |
|
| 253 |
+ end |
|
| 238 | 254 |
end |
| 239 | 255 |
|
| 240 | 256 |
def build_agent |
+ 6
- 0
app/controllers/application_controller.rb
@@ -6,6 +6,12 @@ class ApplicationController < ActionController::Base |
||
| 6 | 6 |
|
| 7 | 7 |
helper :all |
| 8 | 8 |
|
| 9 |
+ def redirect_back(fallback_path, *args) |
|
| 10 |
+ redirect_to :back, *args |
|
| 11 |
+ rescue ActionController::RedirectBackError |
|
| 12 |
+ redirect_to fallback_path, *args |
|
| 13 |
+ end |
|
| 14 |
+ |
|
| 9 | 15 |
protected |
| 10 | 16 |
|
| 11 | 17 |
def configure_permitted_parameters |
+ 4
- 2
app/controllers/events_controller.rb
@@ -24,14 +24,16 @@ class EventsController < ApplicationController |
||
| 24 | 24 |
|
| 25 | 25 |
def reemit |
| 26 | 26 |
@event.reemit! |
| 27 |
- redirect_to :back, :notice => "Event re-emitted" |
|
| 27 |
+ respond_to do |format| |
|
| 28 |
+ format.html { redirect_back event_path(@event), notice: 'Event re-emitted.' }
|
|
| 29 |
+ end |
|
| 28 | 30 |
end |
| 29 | 31 |
|
| 30 | 32 |
def destroy |
| 31 | 33 |
@event.destroy |
| 32 | 34 |
|
| 33 | 35 |
respond_to do |format| |
| 34 |
- format.html { redirect_to events_path }
|
|
| 36 |
+ format.html { redirect_back events_path, notice: 'Event deleted.' }
|
|
| 35 | 37 |
format.json { head :no_content }
|
| 36 | 38 |
end |
| 37 | 39 |
end |
+ 9
- 0
app/controllers/jobs_controller.rb
@@ -48,6 +48,15 @@ class JobsController < ApplicationController |
||
| 48 | 48 |
end |
| 49 | 49 |
end |
| 50 | 50 |
|
| 51 |
+ def destroy_all |
|
| 52 |
+ Delayed::Job.where(locked_at: nil).delete_all |
|
| 53 |
+ |
|
| 54 |
+ respond_to do |format| |
|
| 55 |
+ format.html { redirect_to jobs_path, notice: "All jobs removed." }
|
|
| 56 |
+ format.json { render json: '', status: :ok }
|
|
| 57 |
+ end |
|
| 58 |
+ end |
|
| 59 |
+ |
|
| 51 | 60 |
private |
| 52 | 61 |
|
| 53 | 62 |
def running? |
+ 12
- 0
app/helpers/agent_helper.rb
@@ -37,4 +37,16 @@ module AgentHelper |
||
| 37 | 37 |
}.join(delimiter).html_safe |
| 38 | 38 |
end |
| 39 | 39 |
end |
| 40 |
+ |
|
| 41 |
+ def agent_dry_run_with_event_mode(agent) |
|
| 42 |
+ case |
|
| 43 |
+ when agent.cannot_receive_events? |
|
| 44 |
+ 'no'.freeze |
|
| 45 |
+ when agent.cannot_be_scheduled? |
|
| 46 |
+ # incoming event is the only trigger for the agent |
|
| 47 |
+ 'yes'.freeze |
|
| 48 |
+ else |
|
| 49 |
+ 'maybe'.freeze |
|
| 50 |
+ end |
|
| 51 |
+ end |
|
| 40 | 52 |
end |
+ 1
- 0
app/helpers/application_helper.rb
@@ -80,6 +80,7 @@ module ApplicationHelper |
||
| 80 | 80 |
end |
| 81 | 81 |
|
| 82 | 82 |
def service_label(service) |
| 83 |
+ return if service.nil? |
|
| 83 | 84 |
content_tag :span, [ |
| 84 | 85 |
omniauth_provider_icon(service.provider), |
| 85 | 86 |
service_label_text(service) |
+ 8
- 6
app/helpers/jobs_helper.rb
@@ -24,12 +24,14 @@ module JobsHelper |
||
| 24 | 24 |
# |
| 25 | 25 |
# Can return nil, or an instance of Agent. |
| 26 | 26 |
def agent_from_job(job) |
| 27 |
- begin |
|
| 28 |
- Agent.find_by_id(YAML.load(job.handler).job_data['arguments'][0]) |
|
| 29 |
- rescue ArgumentError |
|
| 30 |
- # We can get to this point before all of the agents have loaded (usually, |
|
| 31 |
- # in development) |
|
| 32 |
- nil |
|
| 27 |
+ if data = YAML.load(job.handler).try(:job_data) |
|
| 28 |
+ Agent.find_by_id(data['arguments'][0]) |
|
| 29 |
+ else |
|
| 30 |
+ false |
|
| 33 | 31 |
end |
| 32 |
+ rescue ArgumentError |
|
| 33 |
+ # We can get to this point before all of the agents have loaded (usually, |
|
| 34 |
+ # in development) |
|
| 35 |
+ nil |
|
| 34 | 36 |
end |
| 35 | 37 |
end |
+ 16
- 8
app/models/agent.rb
@@ -13,6 +13,7 @@ class Agent < ActiveRecord::Base |
||
| 13 | 13 |
include HasGuid |
| 14 | 14 |
include LiquidDroppable |
| 15 | 15 |
include DryRunnable |
| 16 |
+ include SortableEvents |
|
| 16 | 17 |
|
| 17 | 18 |
markdown_class_attributes :description, :event_description |
| 18 | 19 |
|
@@ -21,7 +22,7 @@ class Agent < ActiveRecord::Base |
||
| 21 | 22 |
SCHEDULES = %w[every_1m every_2m every_5m every_10m every_30m every_1h every_2h every_5h every_12h every_1d every_2d every_7d |
| 22 | 23 |
midnight 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am 11am noon 1pm 2pm 3pm 4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm never] |
| 23 | 24 |
|
| 24 |
- EVENT_RETENTION_SCHEDULES = [["Forever", 0], ["1 day", 1], *([2, 3, 4, 5, 7, 14, 21, 30, 45, 90, 180, 365].map {|n| ["#{n} days", n] })]
|
|
| 25 |
+ EVENT_RETENTION_SCHEDULES = [["Forever", 0], ['1 hour', 1.hour], ['6 hours', 6.hours], ["1 day", 1.day], *([2, 3, 4, 5, 7, 14, 21, 30, 45, 90, 180, 365].map {|n| ["#{n} days", n.days] })]
|
|
| 25 | 26 |
|
| 26 | 27 |
attr_accessible :options, :memory, :name, :type, :schedule, :controller_ids, :control_target_ids, :disabled, :source_ids, :scenario_ids, :keep_events_for, :propagate_immediately, :drop_pending_events |
| 27 | 28 |
|
@@ -104,12 +105,19 @@ class Agent < ActiveRecord::Base |
||
| 104 | 105 |
raise "Implement me in your subclass" |
| 105 | 106 |
end |
| 106 | 107 |
|
| 107 |
- def create_event(attrs) |
|
| 108 |
+ def build_event(event) |
|
| 109 |
+ event = events.build(event) if event.is_a?(Hash) |
|
| 110 |
+ event.agent = self |
|
| 111 |
+ event.user = user |
|
| 112 |
+ event.expires_at ||= new_event_expiration_date |
|
| 113 |
+ event |
|
| 114 |
+ end |
|
| 115 |
+ |
|
| 116 |
+ def create_event(event) |
|
| 108 | 117 |
if can_create_events? |
| 109 |
- events.create!({
|
|
| 110 |
- :user => user, |
|
| 111 |
- :expires_at => new_event_expiration_date |
|
| 112 |
- }.merge(attrs)) |
|
| 118 |
+ event = build_event(event) |
|
| 119 |
+ event.save! |
|
| 120 |
+ event |
|
| 113 | 121 |
else |
| 114 | 122 |
error "This Agent cannot create events!" |
| 115 | 123 |
end |
@@ -130,14 +138,14 @@ class Agent < ActiveRecord::Base |
||
| 130 | 138 |
end |
| 131 | 139 |
|
| 132 | 140 |
def new_event_expiration_date |
| 133 |
- keep_events_for > 0 ? keep_events_for.days.from_now : nil |
|
| 141 |
+ keep_events_for > 0 ? keep_events_for.seconds.from_now : nil |
|
| 134 | 142 |
end |
| 135 | 143 |
|
| 136 | 144 |
def update_event_expirations! |
| 137 | 145 |
if keep_events_for == 0 |
| 138 | 146 |
events.update_all :expires_at => nil |
| 139 | 147 |
else |
| 140 |
- events.update_all "expires_at = " + rdbms_date_add("created_at", "DAY", keep_events_for.to_i)
|
|
| 148 |
+ events.update_all "expires_at = " + rdbms_date_add("created_at", "SECOND", keep_events_for.to_i)
|
|
| 141 | 149 |
end |
| 142 | 150 |
end |
| 143 | 151 |
|
+ 156
- 33
app/models/agents/data_output_agent.rb
@@ -2,25 +2,54 @@ module Agents |
||
| 2 | 2 |
class DataOutputAgent < Agent |
| 3 | 3 |
cannot_be_scheduled! |
| 4 | 4 |
|
| 5 |
- description do <<-MD |
|
| 6 |
- The Agent outputs received events as either RSS or JSON. Use it to output a public or private stream of Huginn data. |
|
| 5 |
+ description do |
|
| 6 |
+ <<-MD |
|
| 7 |
+ The Agent outputs received events as either RSS or JSON. Use it to output a public or private stream of Huginn data. |
|
| 7 | 8 |
|
| 8 |
- This Agent will output data at: |
|
| 9 |
+ This Agent will output data at: |
|
| 9 | 10 |
|
| 10 |
- `https://#{ENV['DOMAIN']}/users/#{user.id}/web_requests/#{id || '<id>'}/:secret.xml`
|
|
| 11 |
+ `https://#{ENV['DOMAIN']}#{Rails.application.routes.url_helpers.web_requests_path(agent_id: ':id', user_id: user_id, secret: ':secret', format: :xml)}`
|
|
| 11 | 12 |
|
| 12 |
- where `:secret` is one of the allowed secrets specified in your options and the extension can be `xml` or `json`. |
|
| 13 |
+ where `:secret` is one of the allowed secrets specified in your options and the extension can be `xml` or `json`. |
|
| 13 | 14 |
|
| 14 |
- You can setup multiple secrets so that you can individually authorize external systems to |
|
| 15 |
- access your Huginn data. |
|
| 15 |
+ You can setup multiple secrets so that you can individually authorize external systems to |
|
| 16 |
+ access your Huginn data. |
|
| 16 | 17 |
|
| 17 |
- Options: |
|
| 18 |
+ Options: |
|
| 19 |
+ |
|
| 20 |
+ * `secrets` - An array of tokens that the requestor must provide for light-weight authentication. |
|
| 21 |
+ * `expected_receive_period_in_days` - How often you expect data to be received by this Agent from other Agents. |
|
| 22 |
+ * `template` - A JSON object representing a mapping between item output keys and incoming event values. Use [Liquid](https://github.com/cantino/huginn/wiki/Formatting-Events-using-Liquid) to format the values. Values of the `link`, `title`, `description` and `icon` keys will be put into the \\<channel\\> section of RSS output. The `item` key will be repeated for every Event. The `pubDate` key for each item will have the creation time of the Event unless given. |
|
| 23 |
+ * `events_to_show` - The number of events to output in RSS or JSON. (default: `40`) |
|
| 24 |
+ * `ttl` - A value for the \\<ttl\\> element in RSS output. (default: `60`) |
|
| 25 |
+ |
|
| 26 |
+ If you'd like to output RSS tags with attributes, such as `enclosure`, use something like the following in your `template`: |
|
| 27 |
+ |
|
| 28 |
+ "enclosure": {
|
|
| 29 |
+ "_attributes": {
|
|
| 30 |
+ "url": "{{media_url}}",
|
|
| 31 |
+ "length": "1234456789", |
|
| 32 |
+ "type": "audio/mpeg" |
|
| 33 |
+ } |
|
| 34 |
+ }, |
|
| 35 |
+ "another_tag": {
|
|
| 36 |
+ "_attributes": {
|
|
| 37 |
+ "key": "value", |
|
| 38 |
+ "another_key": "another_value" |
|
| 39 |
+ }, |
|
| 40 |
+ "_contents": "tag contents (can be an object for nesting)" |
|
| 41 |
+ } |
|
| 42 |
+ |
|
| 43 |
+ # Ordering events in the output |
|
| 44 |
+ |
|
| 45 |
+ #{description_events_order('events in the output')}
|
|
| 46 |
+ |
|
| 47 |
+ # Liquid Templating |
|
| 48 |
+ |
|
| 49 |
+ In Liquid templating, the following variable is available: |
|
| 50 |
+ |
|
| 51 |
+ * `events`: An array of events being output, sorted in the given order, up to `events_to_show` in number. For example, if source events contain a site title in the `site_title` key, you can refer to it in `template.title` by putting `{{events.first.site_title}}`.
|
|
| 18 | 52 |
|
| 19 |
- * `secrets` - An array of tokens that the requestor must provide for light-weight authentication. |
|
| 20 |
- * `expected_receive_period_in_days` - How often you expect data to be received by this Agent from other Agents. |
|
| 21 |
- * `template` - A JSON object representing a mapping between item output keys and incoming event values. Use [Liquid](https://github.com/cantino/huginn/wiki/Formatting-Events-using-Liquid) to format the values. The `item` key will be repeated for every Event. The `pubDate` key for each item will have the creation time of the Event unless given. |
|
| 22 |
- * `events_to_show` - The number of events to output in RSS or JSON. (default: `40`) |
|
| 23 |
- * `ttl` - A value for the <ttl> element in RSS output. (default: `60`) |
|
| 24 | 53 |
MD |
| 25 | 54 |
end |
| 26 | 55 |
|
@@ -34,21 +63,28 @@ module Agents |
||
| 34 | 63 |
"item" => {
|
| 35 | 64 |
"title" => "{{title}}",
|
| 36 | 65 |
"description" => "Secret hovertext: {{hovertext}}",
|
| 37 |
- "link" => "{{url}}",
|
|
| 66 |
+ "link" => "{{url}}"
|
|
| 38 | 67 |
} |
| 39 | 68 |
} |
| 40 | 69 |
} |
| 41 | 70 |
end |
| 42 | 71 |
|
| 43 |
- #"guid" => "", |
|
| 44 |
- # "pubDate" => "" |
|
| 45 |
- |
|
| 46 | 72 |
def working? |
| 47 | 73 |
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs? |
| 48 | 74 |
end |
| 49 | 75 |
|
| 50 | 76 |
def validate_options |
| 51 |
- unless options['secrets'].is_a?(Array) && options['secrets'].length > 0 |
|
| 77 |
+ if options['secrets'].is_a?(Array) && options['secrets'].length > 0 |
|
| 78 |
+ options['secrets'].each do |secret| |
|
| 79 |
+ case secret |
|
| 80 |
+ when %r{[/.]}
|
|
| 81 |
+ errors.add(:base, "secret may not contain a slash or dot") |
|
| 82 |
+ when String |
|
| 83 |
+ else |
|
| 84 |
+ errors.add(:base, "secret must be a string") |
|
| 85 |
+ end |
|
| 86 |
+ end |
|
| 87 |
+ else |
|
| 52 | 88 |
errors.add(:base, "Please specify one or more secrets for 'authenticating' incoming feed requests") |
| 53 | 89 |
end |
| 54 | 90 |
|
@@ -77,15 +113,40 @@ module Agents |
||
| 77 | 113 |
interpolated['template']['link'].presence || "https://#{ENV['DOMAIN']}"
|
| 78 | 114 |
end |
| 79 | 115 |
|
| 116 |
+ def feed_url(options = {})
|
|
| 117 |
+ feed_link + Rails.application.routes.url_helpers. |
|
| 118 |
+ web_requests_path(agent_id: id || ':id', |
|
| 119 |
+ user_id: user_id, |
|
| 120 |
+ secret: options[:secret], |
|
| 121 |
+ format: options[:format]) |
|
| 122 |
+ end |
|
| 123 |
+ |
|
| 124 |
+ def feed_icon |
|
| 125 |
+ interpolated['template']['icon'].presence || feed_link + '/favicon.ico' |
|
| 126 |
+ end |
|
| 127 |
+ |
|
| 80 | 128 |
def feed_description |
| 81 | 129 |
interpolated['template']['description'].presence || "A feed of Events received by the '#{name}' Huginn Agent"
|
| 82 | 130 |
end |
| 83 | 131 |
|
| 84 | 132 |
def receive_web_request(params, method, format) |
| 85 |
- if interpolated['secrets'].include?(params['secret']) |
|
| 86 |
- items = received_events.order('id desc').limit(events_to_show).map do |event|
|
|
| 133 |
+ unless interpolated['secrets'].include?(params['secret']) |
|
| 134 |
+ if format =~ /json/ |
|
| 135 |
+ return [{ error: "Not Authorized" }, 401]
|
|
| 136 |
+ else |
|
| 137 |
+ return ["Not Authorized", 401] |
|
| 138 |
+ end |
|
| 139 |
+ end |
|
| 140 |
+ |
|
| 141 |
+ source_events = sort_events(received_events.order(id: :desc).limit(events_to_show).to_a) |
|
| 142 |
+ |
|
| 143 |
+ interpolation_context.stack do |
|
| 144 |
+ interpolation_context['events'] = source_events |
|
| 145 |
+ |
|
| 146 |
+ items = source_events.map do |event| |
|
| 87 | 147 |
interpolated = interpolate_options(options['template']['item'], event) |
| 88 |
- interpolated['guid'] = event.id |
|
| 148 |
+ interpolated['guid'] = {'_attributes' => {'isPermaLink' => 'false'},
|
|
| 149 |
+ '_contents' => interpolated['guid'].presence || event.id} |
|
| 89 | 150 |
date_string = interpolated['pubDate'].to_s |
| 90 | 151 |
date = |
| 91 | 152 |
begin |
@@ -103,25 +164,27 @@ module Agents |
||
| 103 | 164 |
'title' => feed_title, |
| 104 | 165 |
'description' => feed_description, |
| 105 | 166 |
'pubDate' => Time.now, |
| 106 |
- 'items' => items |
|
| 167 |
+ 'items' => simplify_item_for_json(items) |
|
| 107 | 168 |
} |
| 108 | 169 |
|
| 109 | 170 |
return [content, 200] |
| 110 | 171 |
else |
| 111 | 172 |
content = Utils.unindent(<<-XML) |
| 112 | 173 |
<?xml version="1.0" encoding="UTF-8" ?> |
| 113 |
- <rss version="2.0"> |
|
| 174 |
+ <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> |
|
| 114 | 175 |
<channel> |
| 115 |
- <title>#{feed_title.encode(:xml => :text)}</title>
|
|
| 116 |
- <description>#{feed_description.encode(:xml => :text)}</description>
|
|
| 117 |
- <link>#{feed_link.encode(:xml => :text)}</link>
|
|
| 118 |
- <lastBuildDate>#{Time.now.rfc2822.to_s.encode(:xml => :text)}</lastBuildDate>
|
|
| 119 |
- <pubDate>#{Time.now.rfc2822.to_s.encode(:xml => :text)}</pubDate>
|
|
| 176 |
+ <atom:link href=#{feed_url(secret: params['secret'], format: :xml).encode(xml: :attr)} rel="self" type="application/rss+xml" />
|
|
| 177 |
+ <atom:icon>#{feed_icon.encode(xml: :text)}</atom:icon>
|
|
| 178 |
+ <title>#{feed_title.encode(xml: :text)}</title>
|
|
| 179 |
+ <description>#{feed_description.encode(xml: :text)}</description>
|
|
| 180 |
+ <link>#{feed_link.encode(xml: :text)}</link>
|
|
| 181 |
+ <lastBuildDate>#{Time.now.rfc2822.to_s.encode(xml: :text)}</lastBuildDate>
|
|
| 182 |
+ <pubDate>#{Time.now.rfc2822.to_s.encode(xml: :text)}</pubDate>
|
|
| 120 | 183 |
<ttl>#{feed_ttl}</ttl>
|
| 121 | 184 |
|
| 122 | 185 |
XML |
| 123 | 186 |
|
| 124 |
- content += items.to_xml(:skip_types => true, :root => "items", :skip_instruct => true, :indent => 1).gsub(/^<\/?items>/, '').strip |
|
| 187 |
+ content += simplify_item_for_xml(items).to_xml(skip_types: true, root: "items", skip_instruct: true, indent: 1).gsub(/^<\/?items>/, '').strip |
|
| 125 | 188 |
|
| 126 | 189 |
content += Utils.unindent(<<-XML) |
| 127 | 190 |
</channel> |
@@ -130,13 +193,73 @@ module Agents |
||
| 130 | 193 |
|
| 131 | 194 |
return [content, 200, 'text/xml'] |
| 132 | 195 |
end |
| 133 |
- else |
|
| 134 |
- if format =~ /json/ |
|
| 135 |
- return [{ :error => "Not Authorized" }, 401]
|
|
| 196 |
+ end |
|
| 197 |
+ end |
|
| 198 |
+ |
|
| 199 |
+ private |
|
| 200 |
+ |
|
| 201 |
+ class XMLNode |
|
| 202 |
+ def initialize(tag_name, attributes, contents) |
|
| 203 |
+ @tag_name, @attributes, @contents = tag_name, attributes, contents |
|
| 204 |
+ end |
|
| 205 |
+ |
|
| 206 |
+ def to_xml(options) |
|
| 207 |
+ if @contents.is_a?(Hash) |
|
| 208 |
+ options[:builder].tag! @tag_name, @attributes do |
|
| 209 |
+ @contents.each { |key, value| ActiveSupport::XmlMini.to_tag(key, value, options.merge(skip_instruct: true)) }
|
|
| 210 |
+ end |
|
| 136 | 211 |
else |
| 137 |
- return ["Not Authorized", 401] |
|
| 212 |
+ options[:builder].tag! @tag_name, @attributes, @contents |
|
| 138 | 213 |
end |
| 139 | 214 |
end |
| 140 | 215 |
end |
| 216 |
+ |
|
| 217 |
+ def simplify_item_for_xml(item) |
|
| 218 |
+ if item.is_a?(Hash) |
|
| 219 |
+ item.each.with_object({}) do |(key, value), memo|
|
|
| 220 |
+ if value.is_a?(Hash) |
|
| 221 |
+ if value.key?('_attributes') || value.key?('_contents')
|
|
| 222 |
+ memo[key] = XMLNode.new(key, value['_attributes'], simplify_item_for_xml(value['_contents'])) |
|
| 223 |
+ else |
|
| 224 |
+ memo[key] = simplify_item_for_xml(value) |
|
| 225 |
+ end |
|
| 226 |
+ else |
|
| 227 |
+ memo[key] = value |
|
| 228 |
+ end |
|
| 229 |
+ end |
|
| 230 |
+ elsif item.is_a?(Array) |
|
| 231 |
+ item.map { |value| simplify_item_for_xml(value) }
|
|
| 232 |
+ else |
|
| 233 |
+ item |
|
| 234 |
+ end |
|
| 235 |
+ end |
|
| 236 |
+ |
|
| 237 |
+ def simplify_item_for_json(item) |
|
| 238 |
+ if item.is_a?(Hash) |
|
| 239 |
+ item.each.with_object({}) do |(key, value), memo|
|
|
| 240 |
+ if value.is_a?(Hash) |
|
| 241 |
+ if value.key?('_attributes') || value.key?('_contents')
|
|
| 242 |
+ contents = if value['_contents'] && value['_contents'].is_a?(Hash) |
|
| 243 |
+ simplify_item_for_json(value['_contents']) |
|
| 244 |
+ elsif value['_contents'] |
|
| 245 |
+ { "contents" => value['_contents'] }
|
|
| 246 |
+ else |
|
| 247 |
+ {}
|
|
| 248 |
+ end |
|
| 249 |
+ |
|
| 250 |
+ memo[key] = contents.merge(value['_attributes'] || {})
|
|
| 251 |
+ else |
|
| 252 |
+ memo[key] = simplify_item_for_json(value) |
|
| 253 |
+ end |
|
| 254 |
+ else |
|
| 255 |
+ memo[key] = value |
|
| 256 |
+ end |
|
| 257 |
+ end |
|
| 258 |
+ elsif item.is_a?(Array) |
|
| 259 |
+ item.map { |value| simplify_item_for_json(value) }
|
|
| 260 |
+ else |
|
| 261 |
+ item |
|
| 262 |
+ end |
|
| 263 |
+ end |
|
| 141 | 264 |
end |
| 142 | 265 |
end |
+ 2
- 2
app/models/agents/de_duplication_agent.rb
@@ -29,10 +29,10 @@ module Agents |
||
| 29 | 29 |
form_configurable :lookback |
| 30 | 30 |
form_configurable :expected_update_period_in_days |
| 31 | 31 |
|
| 32 |
- before_create :initialize_memory |
|
| 32 |
+ after_initialize :initialize_memory |
|
| 33 | 33 |
|
| 34 | 34 |
def initialize_memory |
| 35 |
- memory['properties'] = [] |
|
| 35 |
+ memory['properties'] ||= [] |
|
| 36 | 36 |
end |
| 37 | 37 |
|
| 38 | 38 |
def validate_options |
+ 1
- 1
app/models/agents/email_agent.rb
@@ -33,8 +33,8 @@ module Agents |
||
| 33 | 33 |
|
| 34 | 34 |
def receive(incoming_events) |
| 35 | 35 |
incoming_events.each do |event| |
| 36 |
- log "Sending digest mail to #{user.email} with event #{event.id}"
|
|
| 37 | 36 |
recipients(event.payload).each do |recipient| |
| 37 |
+ log "Sending digest mail to #{recipient} with event #{event.id}"
|
|
| 38 | 38 |
SystemMailer.send_message(:to => recipient, :subject => interpolated(event)['subject'], :headline => interpolated(event)['headline'], :body => interpolated(event)['body'], :groups => [present(event.payload)]).deliver_later |
| 39 | 39 |
end |
| 40 | 40 |
end |
+ 1
- 1
app/models/agents/email_digest_agent.rb
@@ -40,8 +40,8 @@ module Agents |
||
| 40 | 40 |
if self.memory['queue'] && self.memory['queue'].length > 0 |
| 41 | 41 |
ids = self.memory['events'].join(",")
|
| 42 | 42 |
groups = self.memory['queue'].map { |payload| present(payload) }
|
| 43 |
- log "Sending digest mail to #{user.email} with events [#{ids}]"
|
|
| 44 | 43 |
recipients.each do |recipient| |
| 44 |
+ log "Sending digest mail to #{recipient} with events [#{ids}]"
|
|
| 45 | 45 |
SystemMailer.send_message(:to => recipient, :subject => interpolated['subject'], :headline => interpolated['headline'], :groups => groups).deliver_later |
| 46 | 46 |
end |
| 47 | 47 |
self.memory['queue'] = [] |
+ 1
- 1
app/models/agents/imap_folder_agent.rb
@@ -509,7 +509,7 @@ module Agents |
||
| 509 | 509 |
module Scrubbed |
| 510 | 510 |
def scrubbed(method) |
| 511 | 511 |
(@scrubbed ||= {})[method.to_sym] ||=
|
| 512 |
- __send__(method).scrub { |bytes| "<#{bytes.unpack('H*')[0]}>" }
|
|
| 512 |
+ __send__(method).try(:scrub) { |bytes| "<#{bytes.unpack('H*')[0]}>" }
|
|
| 513 | 513 |
end |
| 514 | 514 |
end |
| 515 | 515 |
|
+ 67
- 33
app/models/agents/rss_agent.rb
@@ -6,25 +6,38 @@ module Agents |
||
| 6 | 6 |
include WebRequestConcern |
| 7 | 7 |
|
| 8 | 8 |
cannot_receive_events! |
| 9 |
+ can_dry_run! |
|
| 9 | 10 |
default_schedule "every_1d" |
| 10 | 11 |
|
| 11 |
- description do <<-MD |
|
| 12 |
- This Agent consumes RSS feeds and emits events when they change. |
|
| 12 |
+ DEFAULT_EVENTS_ORDER = [['{{date_published}}', 'time'], ['{{last_updated}}', 'time']]
|
|
| 13 | 13 |
|
| 14 |
- This Agent is fairly simple, using [feed-normalizer](https://github.com/aasmith/feed-normalizer) as a base. For complex feeds |
|
| 15 |
- with additional field types, we recommend using a WebsiteAgent. See [this example](https://github.com/cantino/huginn/wiki/Agent-configuration-examples#itunes-trailers). |
|
| 14 |
+ description do |
|
| 15 |
+ <<-MD |
|
| 16 |
+ This Agent consumes RSS feeds and emits events when they change. |
|
| 16 | 17 |
|
| 17 |
- If you want to *output* an RSS feed, use the DataOutputAgent. |
|
| 18 |
+ This Agent is fairly simple, using [feed-normalizer](https://github.com/aasmith/feed-normalizer) as a base. For complex feeds |
|
| 19 |
+ with additional field types, we recommend using a WebsiteAgent. See [this example](https://github.com/cantino/huginn/wiki/Agent-configuration-examples#itunes-trailers). |
|
| 18 | 20 |
|
| 19 |
- Options: |
|
| 21 |
+ If you want to *output* an RSS feed, use the DataOutputAgent. |
|
| 20 | 22 |
|
| 21 |
- * `url` - The URL of the RSS feed. |
|
| 22 |
- * `clean` - Attempt to use [feed-normalizer](https://github.com/aasmith/feed-normalizer)'s' `clean!` method to cleanup HTML in the feed. Set to `true` to use. |
|
| 23 |
- * `expected_update_period_in_days` - How often you expect this RSS feed to change. If more than this amount of time passes without an update, the Agent will mark itself as not working. |
|
| 24 |
- * `headers` - When present, it should be a hash of headers to send with the request. |
|
| 25 |
- * `basic_auth` - Specify HTTP basic auth parameters: `"username:password"`, or `["username", "password"]`. |
|
| 26 |
- * `disable_ssl_verification` - Set to `true` to disable ssl verification. |
|
| 27 |
- * `user_agent` - A custom User-Agent name (default: "Faraday v#{Faraday::VERSION}").
|
|
| 23 |
+ Options: |
|
| 24 |
+ |
|
| 25 |
+ * `url` - The URL of the RSS feed (an array of URLs can also be used; items with identical guids across feeds will be considered duplicates). |
|
| 26 |
+ * `clean` - Attempt to use [feed-normalizer](https://github.com/aasmith/feed-normalizer)'s' `clean!` method to cleanup HTML in the feed. Set to `true` to use. |
|
| 27 |
+ * `expected_update_period_in_days` - How often you expect this RSS feed to change. If more than this amount of time passes without an update, the Agent will mark itself as not working. |
|
| 28 |
+ * `headers` - When present, it should be a hash of headers to send with the request. |
|
| 29 |
+ * `basic_auth` - Specify HTTP basic auth parameters: `"username:password"`, or `["username", "password"]`. |
|
| 30 |
+ * `disable_ssl_verification` - Set to `true` to disable ssl verification. |
|
| 31 |
+ * `disable_url_encoding` - Set to `true` to disable url encoding. |
|
| 32 |
+ * `force_encoding` - Set `force_encoding` to an encoding name if the website is known to respond with a missing, invalid or wrong charset in the Content-Type header. Note that a text content without a charset is taken as encoded in UTF-8 (not ISO-8859-1). |
|
| 33 |
+ * `user_agent` - A custom User-Agent name (default: "Faraday v#{Faraday::VERSION}").
|
|
| 34 |
+ * `max_events_per_run` - Limit number of events created (items parsed) per run for feed. |
|
| 35 |
+ |
|
| 36 |
+ # Ordering Events |
|
| 37 |
+ |
|
| 38 |
+ #{description_events_order}
|
|
| 39 |
+ |
|
| 40 |
+ In this Agent, the default value for `events_order` is `#{DEFAULT_EVENTS_ORDER.to_json}`.
|
|
| 28 | 41 |
MD |
| 29 | 42 |
end |
| 30 | 43 |
|
@@ -66,40 +79,44 @@ module Agents |
||
| 66 | 79 |
end |
| 67 | 80 |
|
| 68 | 81 |
validate_web_request_options! |
| 82 |
+ validate_events_order |
|
| 83 |
+ end |
|
| 84 |
+ |
|
| 85 |
+ def events_order |
|
| 86 |
+ super.presence || DEFAULT_EVENTS_ORDER |
|
| 69 | 87 |
end |
| 70 | 88 |
|
| 71 | 89 |
def check |
| 72 |
- response = faraday.get(interpolated['url']) |
|
| 90 |
+ Array(interpolated['url']).each do |url| |
|
| 91 |
+ check_url(url) |
|
| 92 |
+ end |
|
| 93 |
+ end |
|
| 94 |
+ |
|
| 95 |
+ protected |
|
| 96 |
+ |
|
| 97 |
+ def check_url(url) |
|
| 98 |
+ response = faraday.get(url) |
|
| 73 | 99 |
if response.success? |
| 74 | 100 |
feed = FeedNormalizer::FeedNormalizer.parse(response.body) |
| 75 |
- feed.clean! if interpolated['clean'] == 'true' |
|
| 101 |
+ feed.clean! if boolify(interpolated['clean']) |
|
| 102 |
+ max_events = (interpolated['max_events_per_run'].presence || 0).to_i |
|
| 76 | 103 |
created_event_count = 0 |
| 77 |
- feed.entries.each do |entry| |
|
| 78 |
- entry_id = get_entry_id(entry) |
|
| 104 |
+ sort_events(feed_to_events(feed)).each.with_index do |event, index| |
|
| 105 |
+ break if max_events && max_events > 0 && index >= max_events |
|
| 106 |
+ entry_id = event.payload[:id] |
|
| 79 | 107 |
if check_and_track(entry_id) |
| 80 | 108 |
created_event_count += 1 |
| 81 |
- create_event(payload: {
|
|
| 82 |
- id: entry_id, |
|
| 83 |
- date_published: entry.date_published, |
|
| 84 |
- last_updated: entry.last_updated, |
|
| 85 |
- url: entry.url, |
|
| 86 |
- urls: entry.urls, |
|
| 87 |
- description: entry.description, |
|
| 88 |
- content: entry.content, |
|
| 89 |
- title: entry.title, |
|
| 90 |
- authors: entry.authors, |
|
| 91 |
- categories: entry.categories |
|
| 92 |
- }) |
|
| 109 |
+ create_event(event) |
|
| 93 | 110 |
end |
| 94 | 111 |
end |
| 95 |
- log "Fetched #{interpolated['url']} and created #{created_event_count} event(s)."
|
|
| 112 |
+ log "Fetched #{url} and created #{created_event_count} event(s)."
|
|
| 96 | 113 |
else |
| 97 |
- error "Failed to fetch #{interpolated['url']}: #{response.inspect}"
|
|
| 114 |
+ error "Failed to fetch #{url}: #{response.inspect}"
|
|
| 98 | 115 |
end |
| 116 |
+ rescue => e |
|
| 117 |
+ error "Failed to fetch #{url} with message '#{e.message}': #{e.backtrace}"
|
|
| 99 | 118 |
end |
| 100 | 119 |
|
| 101 |
- protected |
|
| 102 |
- |
|
| 103 | 120 |
def get_entry_id(entry) |
| 104 | 121 |
entry.id.presence || Digest::MD5.hexdigest(entry.content) |
| 105 | 122 |
end |
@@ -114,5 +131,22 @@ module Agents |
||
| 114 | 131 |
true |
| 115 | 132 |
end |
| 116 | 133 |
end |
| 134 |
+ |
|
| 135 |
+ def feed_to_events(feed) |
|
| 136 |
+ feed.entries.map { |entry|
|
|
| 137 |
+ Event.new(payload: {
|
|
| 138 |
+ id: get_entry_id(entry), |
|
| 139 |
+ date_published: entry.date_published, |
|
| 140 |
+ last_updated: entry.last_updated, |
|
| 141 |
+ url: entry.url, |
|
| 142 |
+ urls: entry.urls, |
|
| 143 |
+ description: entry.description, |
|
| 144 |
+ content: entry.content, |
|
| 145 |
+ title: entry.title, |
|
| 146 |
+ authors: entry.authors, |
|
| 147 |
+ categories: entry.categories |
|
| 148 |
+ }) |
|
| 149 |
+ } |
|
| 150 |
+ end |
|
| 117 | 151 |
end |
| 118 | 152 |
end |
+ 2
- 2
app/models/agents/slack_agent.rb
@@ -72,9 +72,9 @@ module Agents |
||
| 72 | 72 |
incoming_events.each do |event| |
| 73 | 73 |
opts = interpolated(event) |
| 74 | 74 |
if /^:/.match(opts[:icon]) |
| 75 |
- slack_notifier.ping opts[:message], channel: opts[:channel], username: opts[:username], icon_emoji: opts[:icon] |
|
| 75 |
+ slack_notifier.ping opts[:message], channel: opts[:channel], username: opts[:username], icon_emoji: opts[:icon], unfurl_links: opts[:unfurl_links] |
|
| 76 | 76 |
else |
| 77 |
- slack_notifier.ping opts[:message], channel: opts[:channel], username: opts[:username], icon_url: opts[:icon] |
|
| 77 |
+ slack_notifier.ping opts[:message], channel: opts[:channel], username: opts[:username], icon_url: opts[:icon], unfurl_links: opts[:unfurl_links] |
|
| 78 | 78 |
end |
| 79 | 79 |
end |
| 80 | 80 |
end |
+ 8
- 1
app/models/agents/twitter_user_agent.rb
@@ -14,6 +14,8 @@ module Agents |
||
| 14 | 14 |
You must also provide the `username` of the Twitter user to monitor. |
| 15 | 15 |
|
| 16 | 16 |
Set `include_retweets` to `false` to not include retweets (default: `true`) |
| 17 |
+ |
|
| 18 |
+ Set `exclude_replies` to `true` to exclude replies (default: `false`) |
|
| 17 | 19 |
|
| 18 | 20 |
Set `expected_update_period_in_days` to the maximum amount of time that you'd expect to pass between Events being created by this Agent. |
| 19 | 21 |
|
@@ -54,6 +56,7 @@ module Agents |
||
| 54 | 56 |
{
|
| 55 | 57 |
'username' => 'tectonic', |
| 56 | 58 |
'include_retweets' => 'true', |
| 59 |
+ 'exclude_replies' => 'false', |
|
| 57 | 60 |
'expected_update_period_in_days' => '2' |
| 58 | 61 |
} |
| 59 | 62 |
end |
@@ -82,10 +85,14 @@ module Agents |
||
| 82 | 85 |
def include_retweets? |
| 83 | 86 |
interpolated[:include_retweets] != "false" |
| 84 | 87 |
end |
| 88 |
+ |
|
| 89 |
+ def exclude_replies? |
|
| 90 |
+ boolify(interpolated[:exclude_replies]) || false |
|
| 91 |
+ end |
|
| 85 | 92 |
|
| 86 | 93 |
def check |
| 87 | 94 |
since_id = memory['since_id'] || nil |
| 88 |
- opts = {:count => 200, :include_rts => include_retweets?, :exclude_replies => false, :include_entities => true, :contributor_details => true}
|
|
| 95 |
+ opts = {:count => 200, :include_rts => include_retweets?, :exclude_replies => exclude_replies?, :include_entities => true, :contributor_details => true}
|
|
| 89 | 96 |
opts.merge! :since_id => since_id unless since_id.nil? |
| 90 | 97 |
|
| 91 | 98 |
# http://rdoc.info/gems/twitter/Twitter/REST/Timelines#user_timeline-instance_method |
+ 152
- 40
app/models/agents/website_agent.rb
@@ -6,6 +6,7 @@ module Agents |
||
| 6 | 6 |
include WebRequestConcern |
| 7 | 7 |
|
| 8 | 8 |
can_dry_run! |
| 9 |
+ can_order_created_events! |
|
| 9 | 10 |
|
| 10 | 11 |
default_schedule "every_12h" |
| 11 | 12 |
|
@@ -19,11 +20,19 @@ module Agents |
||
| 19 | 20 |
|
| 20 | 21 |
`url` can be a single url, or an array of urls (for example, for multiple pages with the exact same structure but different content to scrape) |
| 21 | 22 |
|
| 23 |
+ The WebsiteAgent can also scrape based on incoming events. It will scrape the url contained in the `url` key of the incoming event payload, or if you set `url_from_event` it is used as a Liquid template to generate the url to access. If you specify `merge` as the `mode`, it will retain the old payload and update it with the new values. |
|
| 24 |
+ |
|
| 25 |
+ # Supported Document Types |
|
| 26 |
+ |
|
| 22 | 27 |
The `type` value can be `xml`, `html`, `json`, or `text`. |
| 23 | 28 |
|
| 24 | 29 |
To tell the Agent how to parse the content, specify `extract` as a hash with keys naming the extractions and values of hashes. |
| 25 | 30 |
|
| 26 |
- When parsing HTML or XML, these sub-hashes specify how each extraction should be done. The Agent first selects a node set from the document for each extraction key by evaluating either a CSS selector in `css` or an XPath expression in `xpath`. It then evaluates an XPath expression in `value` on each node in the node set, converting the result into string. Here's an example: |
|
| 31 |
+ Note that for all of the formats, whatever you extract MUST have the same number of matches for each extractor. E.g., if you're extracting rows, all extractors must match all rows. For generating CSS selectors, something like [SelectorGadget](http://selectorgadget.com) may be helpful. |
|
| 32 |
+ |
|
| 33 |
+ # Scraping HTML and XML |
|
| 34 |
+ |
|
| 35 |
+ When parsing HTML or XML, these sub-hashes specify how each extraction should be done. The Agent first selects a node set from the document for each extraction key by evaluating either a CSS selector in `css` or an XPath expression in `xpath`. It then evaluates an XPath expression in `value` (default: `.`) on each node in the node set, converting the result into string. Here's an example: |
|
| 27 | 36 |
|
| 28 | 37 |
"extract": {
|
| 29 | 38 |
"url": { "css": "#comic img", "value": "@src" },
|
@@ -31,7 +40,13 @@ module Agents |
||
| 31 | 40 |
"body_text": { "css": "div.main", "value": ".//text()" }
|
| 32 | 41 |
} |
| 33 | 42 |
|
| 34 |
- "@_attr_" is the XPath expression to extract the value of an attribute named _attr_ from a node, and ".//text()" is to extract all the enclosed texts. You can also use [XPath functions](http://www.w3.org/TR/xpath/#section-String-Functions) like `normalize-space` to strip and squeeze whitespace, `substring-after` to extract part of a text, and `translate` to remove comma from a formatted number, etc. Note that these functions take a string, not a node set, so what you may think would be written as `normalize-space(.//text())` should actually be `normalize-space(.)`. |
|
| 43 |
+ "@_attr_" is the XPath expression to extract the value of an attribute named _attr_ from a node, and ".//text()" is to extract all the enclosed texts. To extract the innerHTML, use "./node()"; and to extract the outer HTML, use ".". |
|
| 44 |
+ |
|
| 45 |
+ You can also use [XPath functions](http://www.w3.org/TR/xpath/#section-String-Functions) like `normalize-space` to strip and squeeze whitespace, `substring-after` to extract part of a text, and `translate` to remove comma from a formatted number, etc. Note that these functions take a string, not a node set, so what you may think would be written as `normalize-space(.//text())` should actually be `normalize-space(.)`. |
|
| 46 |
+ |
|
| 47 |
+ Beware that when parsing an XML document (i.e. `type` is `xml`) using `xpath` expressions all namespaces are stripped from the document unless a toplevel option `use_namespaces` is set to true. |
|
| 48 |
+ |
|
| 49 |
+ # Scraping JSON |
|
| 35 | 50 |
|
| 36 | 51 |
When parsing JSON, these sub-hashes specify [JSONPaths](http://goessner.net/articles/JsonPath/) to the values that you care about. For example: |
| 37 | 52 |
|
@@ -40,6 +55,8 @@ module Agents |
||
| 40 | 55 |
"description": { "path": "results.data[*].description" }
|
| 41 | 56 |
} |
| 42 | 57 |
|
| 58 |
+ # Scraping Text |
|
| 59 |
+ |
|
| 43 | 60 |
When parsing text, each sub-hash should contain a `regexp` and `index`. Output text is matched against the regular expression repeatedly from the beginning through to the end, collecting a captured group specified by `index` in each match. Each index should be either an integer or a string name which corresponds to <code>(?<<em>name</em>>...)</code>. For example, to parse lines of <code><em>word</em>: <em>definition</em></code>, the following should work: |
| 44 | 61 |
|
| 45 | 62 |
"extract": {
|
@@ -62,7 +79,7 @@ module Agents |
||
| 62 | 79 |
|
| 63 | 80 |
Beware that `.` does not match the newline character (LF) unless the `m` flag is in effect, and `^`/`$` basically match every line beginning/end. See [this document](http://ruby-doc.org/core-#{RUBY_VERSION}/doc/regexp_rdoc.html) to learn the regular expression variant used in this service.
|
| 64 | 81 |
|
| 65 |
- Note that for all of the formats, whatever you extract MUST have the same number of matches for each extractor. E.g., if you're extracting rows, all extractors must match all rows. For generating CSS selectors, something like [SelectorGadget](http://selectorgadget.com) may be helpful. |
|
| 82 |
+ # General Options |
|
| 66 | 83 |
|
| 67 | 84 |
Can be configured to use HTTP basic auth by including the `basic_auth` parameter with `"username:password"`, or `["username", "password"]`. |
| 68 | 85 |
|
@@ -70,7 +87,7 @@ module Agents |
||
| 70 | 87 |
|
| 71 | 88 |
Set `uniqueness_look_back` to limit the number of events checked for uniqueness (typically for performance). This defaults to the larger of #{UNIQUENESS_LOOK_BACK} or #{UNIQUENESS_FACTOR}x the number of detected received results.
|
| 72 | 89 |
|
| 73 |
- Set `force_encoding` to an encoding name if the website does not return a Content-Type header with a proper charset. |
|
| 90 |
+ Set `force_encoding` to an encoding name if the website is known to respond with a missing, invalid or wrong charset in the Content-Type header. Note that a text content without a charset is taken as encoded in UTF-8 (not ISO-8859-1). |
|
| 74 | 91 |
|
| 75 | 92 |
Set `user_agent` to a custom User-Agent name if the website does not like the default value (`#{default_user_agent}`).
|
| 76 | 93 |
|
@@ -80,7 +97,7 @@ module Agents |
||
| 80 | 97 |
|
| 81 | 98 |
Set `unzip` to `gzip` to inflate the resource using gzip. |
| 82 | 99 |
|
| 83 |
- The WebsiteAgent can also scrape based on incoming events. It will scrape the url contained in the `url` key of the incoming event payload. If you specify `merge` as the mode, it will retain the old payload and update it with the new values. |
|
| 100 |
+ # Liquid Templating |
|
| 84 | 101 |
|
| 85 | 102 |
In Liquid templating, the following variable is available: |
| 86 | 103 |
|
@@ -89,6 +106,10 @@ module Agents |
||
| 89 | 106 |
* `status`: HTTP status as integer. (Almost always 200) |
| 90 | 107 |
|
| 91 | 108 |
* `headers`: Response headers; for example, `{{ _response_.headers.Content-Type }}` expands to the value of the Content-Type header. Keys are insensitive to cases and -/_.
|
| 109 |
+ |
|
| 110 |
+ # Ordering Events |
|
| 111 |
+ |
|
| 112 |
+ #{description_events_order}
|
|
| 92 | 113 |
MD |
| 93 | 114 |
|
| 94 | 115 |
event_description do |
@@ -100,7 +121,7 @@ module Agents |
||
| 100 | 121 |
end |
| 101 | 122 |
|
| 102 | 123 |
def working? |
| 103 |
- event_created_within?(interpolated['expected_update_period_in_days']) && !recent_error_logs? |
|
| 124 |
+ event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs? |
|
| 104 | 125 |
end |
| 105 | 126 |
|
| 106 | 127 |
def default_options |
@@ -119,10 +140,9 @@ module Agents |
||
| 119 | 140 |
|
| 120 | 141 |
def validate_options |
| 121 | 142 |
# Check for required fields |
| 122 |
- errors.add(:base, "url and expected_update_period_in_days are required") unless options['expected_update_period_in_days'].present? && options['url'].present? |
|
| 123 |
- if !options['extract'].present? && extraction_type != "json" |
|
| 124 |
- errors.add(:base, "extract is required for all types except json") |
|
| 125 |
- end |
|
| 143 |
+ errors.add(:base, "either url or url_from_event is required") unless options['url'].present? || options['url_from_event'].present? |
|
| 144 |
+ errors.add(:base, "expected_update_period_in_days is required") unless options['expected_update_period_in_days'].present? |
|
| 145 |
+ validate_extract_options! |
|
| 126 | 146 |
|
| 127 | 147 |
# Check for optional fields |
| 128 | 148 |
if options['mode'].present? |
@@ -137,20 +157,94 @@ module Agents |
||
| 137 | 157 |
errors.add(:base, "Invalid uniqueness_look_back format") unless is_positive_integer?(options['uniqueness_look_back']) |
| 138 | 158 |
end |
| 139 | 159 |
|
| 140 |
- if (encoding = options['force_encoding']).present? |
|
| 141 |
- case encoding |
|
| 142 |
- when String |
|
| 143 |
- begin |
|
| 144 |
- Encoding.find(encoding) |
|
| 145 |
- rescue ArgumentError |
|
| 146 |
- errors.add(:base, "Unknown encoding: #{encoding.inspect}")
|
|
| 147 |
- end |
|
| 160 |
+ validate_web_request_options! |
|
| 161 |
+ end |
|
| 162 |
+ |
|
| 163 |
+ def validate_extract_options! |
|
| 164 |
+ extraction_type = (extraction_type() rescue extraction_type(options)) |
|
| 165 |
+ case extract = options['extract'] |
|
| 166 |
+ when Hash |
|
| 167 |
+ if extract.each_value.any? { |value| !value.is_a?(Hash) }
|
|
| 168 |
+ errors.add(:base, 'extract must be a hash of hashes.') |
|
| 148 | 169 |
else |
| 149 |
- errors.add(:base, "force_encoding must be a string") |
|
| 170 |
+ case extraction_type |
|
| 171 |
+ when 'html', 'xml' |
|
| 172 |
+ extract.each do |name, details| |
|
| 173 |
+ case details['css'] |
|
| 174 |
+ when String |
|
| 175 |
+ # ok |
|
| 176 |
+ when nil |
|
| 177 |
+ case details['xpath'] |
|
| 178 |
+ when String |
|
| 179 |
+ # ok |
|
| 180 |
+ when nil |
|
| 181 |
+ errors.add(:base, "When type is html or xml, all extractions must have a css or xpath attribute (bad extraction details for #{name.inspect})")
|
|
| 182 |
+ else |
|
| 183 |
+ errors.add(:base, "Wrong type of \"xpath\" value in extraction details for #{name.inspect}")
|
|
| 184 |
+ end |
|
| 185 |
+ else |
|
| 186 |
+ errors.add(:base, "Wrong type of \"css\" value in extraction details for #{name.inspect}")
|
|
| 187 |
+ end |
|
| 188 |
+ |
|
| 189 |
+ case details['value'] |
|
| 190 |
+ when String, nil |
|
| 191 |
+ # ok |
|
| 192 |
+ else |
|
| 193 |
+ errors.add(:base, "Wrong type of \"value\" value in extraction details for #{name.inspect}")
|
|
| 194 |
+ end |
|
| 195 |
+ end |
|
| 196 |
+ when 'json' |
|
| 197 |
+ extract.each do |name, details| |
|
| 198 |
+ case details['path'] |
|
| 199 |
+ when String |
|
| 200 |
+ # ok |
|
| 201 |
+ when nil |
|
| 202 |
+ errors.add(:base, "When type is json, all extractions must have a path attribute (bad extraction details for #{name.inspect})")
|
|
| 203 |
+ else |
|
| 204 |
+ errors.add(:base, "Wrong type of \"path\" value in extraction details for #{name.inspect}")
|
|
| 205 |
+ end |
|
| 206 |
+ end |
|
| 207 |
+ when 'text' |
|
| 208 |
+ extract.each do |name, details| |
|
| 209 |
+ case regexp = details['regexp'] |
|
| 210 |
+ when String |
|
| 211 |
+ begin |
|
| 212 |
+ re = Regexp.new(regexp) |
|
| 213 |
+ rescue => e |
|
| 214 |
+ errors.add(:base, "invalid regexp for #{name.inspect}: #{e.message}")
|
|
| 215 |
+ end |
|
| 216 |
+ when nil |
|
| 217 |
+ errors.add(:base, "When type is text, all extractions must have a regexp attribute (bad extraction details for #{name.inspect})")
|
|
| 218 |
+ else |
|
| 219 |
+ errors.add(:base, "Wrong type of \"regexp\" value in extraction details for #{name.inspect}")
|
|
| 220 |
+ end |
|
| 221 |
+ |
|
| 222 |
+ case index = details['index'] |
|
| 223 |
+ when Integer, /\A\d+\z/ |
|
| 224 |
+ # ok |
|
| 225 |
+ when String |
|
| 226 |
+ if re && !re.names.include?(index) |
|
| 227 |
+ errors.add(:base, "no named capture #{index.inspect} found in regexp for #{name.inspect})")
|
|
| 228 |
+ end |
|
| 229 |
+ when nil |
|
| 230 |
+ errors.add(:base, "When type is text, all extractions must have an index attribute (bad extraction details for #{name.inspect})")
|
|
| 231 |
+ else |
|
| 232 |
+ errors.add(:base, "Wrong type of \"index\" value in extraction details for #{name.inspect}")
|
|
| 233 |
+ end |
|
| 234 |
+ end |
|
| 235 |
+ when /\{/
|
|
| 236 |
+ # Liquid templating |
|
| 237 |
+ else |
|
| 238 |
+ errors.add(:base, "Unknown extraction type #{extraction_type.inspect}")
|
|
| 239 |
+ end |
|
| 150 | 240 |
end |
| 241 |
+ when nil |
|
| 242 |
+ unless extraction_type == 'json' |
|
| 243 |
+ errors.add(:base, 'extract is required for all types except json') |
|
| 244 |
+ end |
|
| 245 |
+ else |
|
| 246 |
+ errors.add(:base, 'extract must be a hash') |
|
| 151 | 247 |
end |
| 152 |
- |
|
| 153 |
- validate_web_request_options! |
|
| 154 | 248 |
end |
| 155 | 249 |
|
| 156 | 250 |
def check |
@@ -166,6 +260,10 @@ module Agents |
||
| 166 | 260 |
end |
| 167 | 261 |
|
| 168 | 262 |
def check_url(url, payload = {})
|
| 263 |
+ unless /\Ahttps?:\/\//i === url |
|
| 264 |
+ error "Ignoring a non-HTTP url: #{url.inspect}"
|
|
| 265 |
+ return |
|
| 266 |
+ end |
|
| 169 | 267 |
log "Fetching #{url}"
|
| 170 | 268 |
response = faraday.get(url) |
| 171 | 269 |
raise "Failed: #{response.inspect}" unless response.success?
|
@@ -173,12 +271,6 @@ module Agents |
||
| 173 | 271 |
interpolation_context.stack {
|
| 174 | 272 |
interpolation_context['_response_'] = ResponseDrop.new(response) |
| 175 | 273 |
body = response.body |
| 176 |
- if (encoding = interpolated['force_encoding']).present? |
|
| 177 |
- body = body.encode(Encoding::UTF_8, encoding) |
|
| 178 |
- end |
|
| 179 |
- if interpolated['unzip'] == "gzip" |
|
| 180 |
- body = ActiveSupport::Gzip.decompress(body) |
|
| 181 |
- end |
|
| 182 | 274 |
doc = parse(body) |
| 183 | 275 |
|
| 184 | 276 |
if extract_full_json? |
@@ -228,8 +320,12 @@ module Agents |
||
| 228 | 320 |
def receive(incoming_events) |
| 229 | 321 |
incoming_events.each do |event| |
| 230 | 322 |
interpolate_with(event) do |
| 231 |
- url_to_scrape = event.payload['url'] |
|
| 232 |
- next unless url_to_scrape =~ /^https?:\/\//i |
|
| 323 |
+ url_to_scrape = |
|
| 324 |
+ if url_template = options['url_from_event'].presence |
|
| 325 |
+ interpolate_string(url_template) |
|
| 326 |
+ else |
|
| 327 |
+ event.payload['url'] |
|
| 328 |
+ end |
|
| 233 | 329 |
check_url(url_to_scrape, |
| 234 | 330 |
interpolated['mode'].to_s == "merge" ? event.payload : {})
|
| 235 | 331 |
end |
@@ -275,7 +371,7 @@ module Agents |
||
| 275 | 371 |
!interpolated['extract'].present? && extraction_type == "json" |
| 276 | 372 |
end |
| 277 | 373 |
|
| 278 |
- def extraction_type |
|
| 374 |
+ def extraction_type(interpolated = interpolated()) |
|
| 279 | 375 |
(interpolated['type'] || begin |
| 280 | 376 |
case interpolated['url'] |
| 281 | 377 |
when /\.(rss|xml)$/i |
@@ -290,14 +386,24 @@ module Agents |
||
| 290 | 386 |
end).to_s |
| 291 | 387 |
end |
| 292 | 388 |
|
| 293 |
- def extract_each(doc, &block) |
|
| 389 |
+ def use_namespaces? |
|
| 390 |
+ if value = interpolated.key?('use_namespaces')
|
|
| 391 |
+ boolify(interpolated['use_namespaces']) |
|
| 392 |
+ else |
|
| 393 |
+ interpolated['extract'].none? { |name, extraction_details|
|
|
| 394 |
+ extraction_details.key?('xpath')
|
|
| 395 |
+ } |
|
| 396 |
+ end |
|
| 397 |
+ end |
|
| 398 |
+ |
|
| 399 |
+ def extract_each(&block) |
|
| 294 | 400 |
interpolated['extract'].each_with_object({}) { |(name, extraction_details), output|
|
| 295 | 401 |
output[name] = block.call(extraction_details) |
| 296 | 402 |
} |
| 297 | 403 |
end |
| 298 | 404 |
|
| 299 | 405 |
def extract_json(doc) |
| 300 |
- extract_each(doc) { |extraction_details|
|
|
| 406 |
+ extract_each { |extraction_details|
|
|
| 301 | 407 |
result = Utils.values_at(doc, extraction_details['path']) |
| 302 | 408 |
log "Extracting #{extraction_type} at #{extraction_details['path']}: #{result}"
|
| 303 | 409 |
result |
@@ -305,11 +411,15 @@ module Agents |
||
| 305 | 411 |
end |
| 306 | 412 |
|
| 307 | 413 |
def extract_text(doc) |
| 308 |
- extract_each(doc) { |extraction_details|
|
|
| 414 |
+ extract_each { |extraction_details|
|
|
| 309 | 415 |
regexp = Regexp.new(extraction_details['regexp']) |
| 416 |
+ case index = extraction_details['index'] |
|
| 417 |
+ when /\A\d+\z/ |
|
| 418 |
+ index = index.to_i |
|
| 419 |
+ end |
|
| 310 | 420 |
result = [] |
| 311 | 421 |
doc.scan(regexp) {
|
| 312 |
- result << Regexp.last_match[extraction_details['index']] |
|
| 422 |
+ result << Regexp.last_match[index] |
|
| 313 | 423 |
} |
| 314 | 424 |
log "Extracting #{extraction_type} at #{regexp}: #{result}"
|
| 315 | 425 |
result |
@@ -317,12 +427,11 @@ module Agents |
||
| 317 | 427 |
end |
| 318 | 428 |
|
| 319 | 429 |
def extract_xml(doc) |
| 320 |
- extract_each(doc) { |extraction_details|
|
|
| 430 |
+ extract_each { |extraction_details|
|
|
| 321 | 431 |
case |
| 322 | 432 |
when css = extraction_details['css'] |
| 323 | 433 |
nodes = doc.css(css) |
| 324 | 434 |
when xpath = extraction_details['xpath'] |
| 325 |
- doc.remove_namespaces! # ignore xmlns, useful when parsing atom feeds |
|
| 326 | 435 |
nodes = doc.xpath(xpath) |
| 327 | 436 |
else |
| 328 | 437 |
raise '"css" or "xpath" is required for HTML or XML extraction' |
@@ -330,7 +439,7 @@ module Agents |
||
| 330 | 439 |
case nodes |
| 331 | 440 |
when Nokogiri::XML::NodeSet |
| 332 | 441 |
result = nodes.map { |node|
|
| 333 |
- case value = node.xpath(extraction_details['value']) |
|
| 442 |
+ case value = node.xpath(extraction_details['value'] || '.') |
|
| 334 | 443 |
when Float |
| 335 | 444 |
# Node#xpath() returns any numeric value as float; |
| 336 | 445 |
# convert it to integer as appropriate. |
@@ -347,9 +456,12 @@ module Agents |
||
| 347 | 456 |
end |
| 348 | 457 |
|
| 349 | 458 |
def parse(data) |
| 350 |
- case extraction_type |
|
| 459 |
+ case type = extraction_type |
|
| 351 | 460 |
when "xml" |
| 352 |
- Nokogiri::XML(data) |
|
| 461 |
+ doc = Nokogiri::XML(data) |
|
| 462 |
+ # ignore xmlns, useful when parsing atom feeds |
|
| 463 |
+ doc.remove_namespaces! unless use_namespaces? |
|
| 464 |
+ doc |
|
| 353 | 465 |
when "json" |
| 354 | 466 |
JSON.parse(data) |
| 355 | 467 |
when "html" |
@@ -357,7 +469,7 @@ module Agents |
||
| 357 | 469 |
when "text" |
| 358 | 470 |
data |
| 359 | 471 |
else |
| 360 |
- raise "Unknown extraction type #{extraction_type}"
|
|
| 472 |
+ raise "Unknown extraction type: #{type}"
|
|
| 361 | 473 |
end |
| 362 | 474 |
end |
| 363 | 475 |
|
+ 3
- 1
app/models/agents/wunderlist_agent.rb
@@ -52,7 +52,9 @@ module Agents |
||
| 52 | 52 |
end |
| 53 | 53 |
end |
| 54 | 54 |
end |
| 55 |
- private |
|
| 55 |
+ |
|
| 56 |
+ private |
|
| 57 |
+ |
|
| 56 | 58 |
def request_guard(&blk) |
| 57 | 59 |
response = yield |
| 58 | 60 |
error("Error during http request: #{response.body}") if response.code > 400
|
+ 30
- 18
app/models/scenario_import.rb
@@ -60,6 +60,7 @@ class ScenarioImport |
||
| 60 | 60 |
description = parsed_data['description'] |
| 61 | 61 |
name = parsed_data['name'] |
| 62 | 62 |
links = parsed_data['links'] |
| 63 |
+ control_links = parsed_data['control_links'] || [] |
|
| 63 | 64 |
tag_fg_color = parsed_data['tag_fg_color'] |
| 64 | 65 |
tag_bg_color = parsed_data['tag_bg_color'] |
| 65 | 66 |
source_url = parsed_data['source_url'].presence || nil |
@@ -87,12 +88,19 @@ class ScenarioImport |
||
| 87 | 88 |
end |
| 88 | 89 |
agent |
| 89 | 90 |
end |
| 91 |
+ |
|
| 90 | 92 |
if success |
| 91 | 93 |
links.each do |link| |
| 92 | 94 |
receiver = created_agents[link['receiver']] |
| 93 | 95 |
source = created_agents[link['source']] |
| 94 | 96 |
receiver.sources << source unless receiver.sources.include?(source) |
| 95 | 97 |
end |
| 98 |
+ |
|
| 99 |
+ control_links.each do |control_link| |
|
| 100 |
+ controller = created_agents[control_link['controller']] |
|
| 101 |
+ control_target = created_agents[control_link['control_target']] |
|
| 102 |
+ controller.control_targets << control_target unless controller.control_targets.include?(control_target) |
|
| 103 |
+ end |
|
| 96 | 104 |
end |
| 97 | 105 |
end |
| 98 | 106 |
|
@@ -142,7 +150,7 @@ class ScenarioImport |
||
| 142 | 150 |
def generate_diff |
| 143 | 151 |
@agent_diffs = (parsed_data['agents'] || []).map.with_index do |agent_data, index| |
| 144 | 152 |
# AgentDiff is defined at the end of this file. |
| 145 |
- agent_diff = AgentDiff.new(agent_data) |
|
| 153 |
+ agent_diff = AgentDiff.new(agent_data, parsed_data['schema_version']) |
|
| 146 | 154 |
if existing_scenario |
| 147 | 155 |
# If this Agent exists already, update the AgentDiff with the local version's information. |
| 148 | 156 |
agent_diff.diff_with! existing_scenario.agents.find_by(:guid => agent_data['guid']) |
@@ -184,14 +192,16 @@ class ScenarioImport |
||
| 184 | 192 |
end |
| 185 | 193 |
end |
| 186 | 194 |
|
| 187 |
- def initialize(agent_data) |
|
| 195 |
+ def initialize(agent_data, schema_version) |
|
| 188 | 196 |
super() |
| 197 |
+ @schema_version = schema_version |
|
| 189 | 198 |
@requires_merge = false |
| 190 | 199 |
self.agent = nil |
| 191 | 200 |
store! agent_data |
| 192 | 201 |
end |
| 193 | 202 |
|
| 194 | 203 |
BASE_FIELDS = %w[name schedule keep_events_for propagate_immediately disabled guid] |
| 204 |
+ FIELDS_REQUIRING_TRANSLATION = %w[keep_events_for] |
|
| 195 | 205 |
|
| 196 | 206 |
def agent_exists? |
| 197 | 207 |
!!agent |
@@ -209,10 +219,27 @@ class ScenarioImport |
||
| 209 | 219 |
self.type = FieldDiff.new(agent_data["type"].split("::").pop)
|
| 210 | 220 |
self.options = FieldDiff.new(agent_data['options'] || {})
|
| 211 | 221 |
BASE_FIELDS.each do |option| |
| 212 |
- self[option] = FieldDiff.new(agent_data[option]) if agent_data.has_key?(option) |
|
| 222 |
+ if agent_data.has_key?(option) |
|
| 223 |
+ value = agent_data[option] |
|
| 224 |
+ value = send(:"translate_#{option}", value) if option.in?(FIELDS_REQUIRING_TRANSLATION)
|
|
| 225 |
+ self[option] = FieldDiff.new(value) |
|
| 226 |
+ end |
|
| 213 | 227 |
end |
| 214 | 228 |
end |
| 215 | 229 |
|
| 230 |
+ def translate_keep_events_for(old_value) |
|
| 231 |
+ if schema_version < 1 |
|
| 232 |
+ # Was stored in days, now is stored in seconds. |
|
| 233 |
+ old_value.to_i.days |
|
| 234 |
+ else |
|
| 235 |
+ old_value |
|
| 236 |
+ end |
|
| 237 |
+ end |
|
| 238 |
+ |
|
| 239 |
+ def schema_version |
|
| 240 |
+ (@schema_version || 0).to_i |
|
| 241 |
+ end |
|
| 242 |
+ |
|
| 216 | 243 |
def diff_with!(agent) |
| 217 | 244 |
return unless agent.present? |
| 218 | 245 |
|
@@ -251,21 +278,6 @@ class ScenarioImport |
||
| 251 | 278 |
yield 'disabled', disabled, boolean if disabled.requires_merge? |
| 252 | 279 |
end |
| 253 | 280 |
|
| 254 |
- # Unfortunately Ruby 1.9's OpenStruct doesn't expose [] and []=. |
|
| 255 |
- unless instance_methods.include?(:[]=) |
|
| 256 |
- def [](key) |
|
| 257 |
- self.send(sanitize key) |
|
| 258 |
- end |
|
| 259 |
- |
|
| 260 |
- def []=(key, val) |
|
| 261 |
- self.send("#{sanitize key}=", val)
|
|
| 262 |
- end |
|
| 263 |
- |
|
| 264 |
- def sanitize(key) |
|
| 265 |
- key.gsub(/[^a-zA-Z0-9_-]/, '') |
|
| 266 |
- end |
|
| 267 |
- end |
|
| 268 |
- |
|
| 269 | 281 |
def agent_instance |
| 270 | 282 |
"Agents::#{self.type.updated}".constantize.new
|
| 271 | 283 |
end |
+ 5
- 1
app/models/user.rb
@@ -18,7 +18,7 @@ class User < ActiveRecord::Base |
||
| 18 | 18 |
validates_presence_of :username |
| 19 | 19 |
validates_uniqueness_of :username |
| 20 | 20 |
validates_format_of :username, :with => /\A[a-zA-Z0-9_-]{3,15}\Z/, :message => "can only contain letters, numbers, underscores, and dashes, and must be between 3 and 15 characters in length."
|
| 21 |
- validates_inclusion_of :invitation_code, :on => :create, :in => INVITATION_CODES, :message => "is not valid" |
|
| 21 |
+ validates_inclusion_of :invitation_code, :on => :create, :in => INVITATION_CODES, :message => "is not valid", if: ->{ User.using_invitation_code? }
|
|
| 22 | 22 |
|
| 23 | 23 |
has_many :user_credentials, :dependent => :destroy, :inverse_of => :user |
| 24 | 24 |
has_many :events, -> { order("events.created_at desc") }, :dependent => :delete_all, :inverse_of => :user
|
@@ -40,4 +40,8 @@ class User < ActiveRecord::Base |
||
| 40 | 40 |
where(conditions).first |
| 41 | 41 |
end |
| 42 | 42 |
end |
| 43 |
+ |
|
| 44 |
+ def self.using_invitation_code? |
|
| 45 |
+ ENV['SKIP_INVITATION_CODE'] != 'true' |
|
| 46 |
+ end |
|
| 43 | 47 |
end |
+ 1
- 1
app/views/agents/_action_menu.html.erb
@@ -7,7 +7,7 @@ |
||
| 7 | 7 |
|
| 8 | 8 |
<% if agent.can_dry_run? %> |
| 9 | 9 |
<li> |
| 10 |
- <%= link_to icon_tag('glyphicon-refresh') + ' Dry Run', '#', 'data-action-url' => dry_run_agent_path(agent), tabindex: "-1", onclick: "Utils.handleDryRunButton(this, '_method=PUT')" %>
|
|
| 10 |
+ <%= link_to icon_tag('glyphicon-refresh') + ' Dry Run', '#', 'data-action-url' => dry_run_agent_path(agent), 'data-with-event-mode' => agent_dry_run_with_event_mode(agent), tabindex: "-1", onclick: "Utils.handleDryRunButton(this)" %>
|
|
| 11 | 11 |
</li> |
| 12 | 12 |
<% end %> |
| 13 | 13 |
|
+ 1
- 1
app/views/agents/_options.erb
@@ -25,6 +25,6 @@ |
||
| 25 | 25 |
<div class="form-group"> |
| 26 | 26 |
<%= submit_tag "Save", :class => "btn btn-primary" %> |
| 27 | 27 |
<% if agent.can_dry_run? %> |
| 28 |
- <%= button_tag class: 'btn btn-default agent-dry-run-button', type: 'button', 'data-action-url' => agent.persisted? ? dry_run_agent_path(agent) : dry_run_agents_path do %><%= icon_tag('glyphicon-refresh') %> Dry Run<% end %>
|
|
| 28 |
+ <%= button_tag class: 'btn btn-default agent-dry-run-button', type: 'button', 'data-action-url' => agent.persisted? ? dry_run_agent_path(agent) : dry_run_agents_path, 'data-with-event-mode' => agent_dry_run_with_event_mode(agent) do %><%= icon_tag('glyphicon-refresh') %> Dry Run<% end %>
|
|
| 29 | 29 |
<% end %> |
| 30 | 30 |
</div> |
+ 8
- 6
app/views/devise/registrations/new.html.erb
@@ -30,13 +30,15 @@ bin/setup_heroku |
||
| 30 | 30 |
</div> |
| 31 | 31 |
<% end %> |
| 32 | 32 |
|
| 33 |
- <div class="form-group"> |
|
| 34 |
- <%= f.label :invitation_code, class: 'col-md-4 control-label' %> |
|
| 35 |
- <div class="col-md-6"> |
|
| 36 |
- <%= f.text_field :invitation_code, class: 'form-control' %> |
|
| 37 |
- <span class="help-inline">We are not yet open to the public. If you have an invitation code, please enter it here.</span> |
|
| 33 |
+ <% if User.using_invitation_code? %> |
|
| 34 |
+ <div class="form-group"> |
|
| 35 |
+ <%= f.label :invitation_code, class: 'col-md-4 control-label' %> |
|
| 36 |
+ <div class="col-md-6"> |
|
| 37 |
+ <%= f.text_field :invitation_code, class: 'form-control' %> |
|
| 38 |
+ <span class="help-inline">We are not yet open to the public. If you have an invitation code, please enter it here.</span> |
|
| 39 |
+ </div> |
|
| 38 | 40 |
</div> |
| 39 |
- </div> |
|
| 41 |
+ <% end %> |
|
| 40 | 42 |
|
| 41 | 43 |
<div class="form-group"> |
| 42 | 44 |
<%= f.label :email, class: 'col-md-4 control-label' %> |
+ 15
- 3
app/views/jobs/index.html.erb
@@ -20,11 +20,19 @@ |
||
| 20 | 20 |
</tr> |
| 21 | 21 |
|
| 22 | 22 |
<% @jobs.each do |job| %> |
| 23 |
- <% agent = agent_from_job(job) %> |
|
| 24 | 23 |
<tr> |
| 25 | 24 |
<td><%= status(job) %></td> |
| 26 |
- <td><%= agent ? link_to(agent.name, agent_path(agent)) : "(deleted)" %></td> |
|
| 27 |
- <td title='<%= job.created_at %>'><%= time_ago_in_words job.created_at %> ago <%= agent ? "for #{agent.user.username}" : '' %></td>
|
|
| 25 |
+ <td><% case agent = agent_from_job(job) |
|
| 26 |
+ when Agent |
|
| 27 |
+ %><%= link_to(agent.name, agent_path(agent)) %><% |
|
| 28 |
+ when false |
|
| 29 |
+ %>(system)<% |
|
| 30 |
+ when nil |
|
| 31 |
+ %>(deleted)<% |
|
| 32 |
+ else |
|
| 33 |
+ %>(unknown)<% |
|
| 34 |
+ end %></td> |
|
| 35 |
+ <td title='<%= job.created_at %>'><%= time_ago_in_words job.created_at %> ago<% if user = agent.try(:user) %> for <%= user.username %><% end %></td> |
|
| 28 | 36 |
<td title='<%= job.run_at %>'> |
| 29 | 37 |
<% if !job.failed_at %> |
| 30 | 38 |
<%= relative_distance_of_time_in_words job.run_at %> |
@@ -71,6 +79,10 @@ |
||
| 71 | 79 |
<%= link_to destroy_failed_jobs_path, class: "btn btn-default", method: :delete do %> |
| 72 | 80 |
<span class="glyphicon glyphicon-trash"></span> Remove failed jobs |
| 73 | 81 |
<% end %> |
| 82 |
+ |
|
| 83 |
+ <%= link_to destroy_all_jobs_path, class: "btn btn-default", method: :delete, data: { confirm: "Are you sure you want to delete ALL pending jobs for all Huginn users?" } do %>
|
|
| 84 |
+ <span class="glyphicon glyphicon-remove"></span> Remove all jobs |
|
| 85 |
+ <% end %> |
|
| 74 | 86 |
</div> |
| 75 | 87 |
</div> |
| 76 | 88 |
</div> |
+ 13
- 0
app/views/layouts/application.html.erb
@@ -9,6 +9,19 @@ |
||
| 9 | 9 |
<%= stylesheet_link_tag "application", :media => "all" %> |
| 10 | 10 |
<%= javascript_include_tag "application" %> |
| 11 | 11 |
<%= csrf_meta_tags %> |
| 12 |
+ <link rel="apple-touch-icon" sizes="57x57" href="/apple-touch-icon-57x57.png"> |
|
| 13 |
+ <link rel="apple-touch-icon" sizes="60x60" href="/apple-touch-icon-60x60.png"> |
|
| 14 |
+ <link rel="apple-touch-icon" sizes="72x72" href="/apple-touch-icon-72x72.png"> |
|
| 15 |
+ <link rel="apple-touch-icon" sizes="76x76" href="/apple-touch-icon-76x76.png"> |
|
| 16 |
+ <link rel="apple-touch-icon" sizes="114x114" href="/apple-touch-icon-114x114.png"> |
|
| 17 |
+ <link rel="apple-touch-icon" sizes="120x120" href="/apple-touch-icon-120x120.png"> |
|
| 18 |
+ <link rel="apple-touch-icon" sizes="144x144" href="/apple-touch-icon-144x144.png"> |
|
| 19 |
+ <link rel="apple-touch-icon" sizes="152x152" href="/apple-touch-icon-152x152.png"> |
|
| 20 |
+ <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon-180x180.png"> |
|
| 21 |
+ <link rel="icon" type="image/vnd.microsoft.icon" href="/favicon.ico" sizes="16x16"> |
|
| 22 |
+ <link rel="icon" type="image/png" href="/android-chrome-48x48.png" sizes="48x48"> |
|
| 23 |
+ <link rel="icon" type="image/png" href="/android-chrome-192x192.png" sizes="192x192"> |
|
| 24 |
+ <link rel="manifest" href="/manifest.json"> |
|
| 12 | 25 |
<%= yield(:ace_editor_script) %> |
| 13 | 26 |
<%= yield(:head) %> |
| 14 | 27 |
</head> |
+ 1
- 1
app/views/scenario_imports/_step_two.html.erb
@@ -14,7 +14,7 @@ |
||
| 14 | 14 |
<span class='glyphicon glyphicon-warning-sign'></span> |
| 15 | 15 |
This Scenario already exists in your system. The import will update your existing |
| 16 | 16 |
<%= scenario_label(@scenario_import.existing_scenario) %> Scenario's title, |
| 17 |
- description and tag colors. Below you can customize how the individual agents get updated. |
|
| 17 |
+ description, and tag colors. Below you can customize how the individual agents get updated. |
|
| 18 | 18 |
</div> |
| 19 | 19 |
<% end %> |
| 20 | 20 |
|
+ 2
- 1
app/views/scenarios/show.html.erb
@@ -16,10 +16,11 @@ |
||
| 16 | 16 |
|
| 17 | 17 |
<div class="btn-group"> |
| 18 | 18 |
<%= link_to icon_tag('glyphicon-chevron-left') + ' Back', scenarios_path, class: "btn btn-default" %>
|
| 19 |
+ <%= link_to icon_tag('glyphicon-plus') + ' New Agent', new_agent_path(scenario_id: @scenario.id), class: "btn btn-default" %>
|
|
| 19 | 20 |
<%= link_to icon_tag('glyphicon-random') + ' View Diagram', scenario_diagram_path(@scenario), class: "btn btn-default" %>
|
| 20 | 21 |
<%= link_to icon_tag('glyphicon-edit') + ' Edit', edit_scenario_path(@scenario), class: "btn btn-default" %>
|
| 21 | 22 |
<% if @scenario.source_url.present? %> |
| 22 |
- <%= link_to icon_tag('glyphicon-plus') + ' Update', new_scenario_imports_path(url: @scenario.source_url), class: "btn btn-default" %>
|
|
| 23 |
+ <%= link_to icon_tag('glyphicon-refresh') + ' Update', new_scenario_imports_path(url: @scenario.source_url), class: "btn btn-default" %>
|
|
| 23 | 24 |
<% end %> |
| 24 | 25 |
<%= link_to icon_tag('glyphicon-share-alt') + ' Share', share_scenario_path(@scenario), class: "btn btn-default" %>
|
| 25 | 26 |
<%= link_to icon_tag('glyphicon-trash') + ' Delete', scenario_path(@scenario), method: :delete, data: { confirm: "This will remove the '#{@scenario.name}' Scenerio from all Agents and delete it. Are you sure?" }, class: "btn btn-default" %>
|
+ 0
- 4
bin/rails
@@ -1,8 +1,4 @@ |
||
| 1 | 1 |
#!/usr/bin/env ruby |
| 2 |
-begin |
|
| 3 |
- load File.expand_path("../spring", __FILE__)
|
|
| 4 |
-rescue LoadError |
|
| 5 |
-end |
|
| 6 | 2 |
APP_PATH = File.expand_path('../../config/application', __FILE__)
|
| 7 | 3 |
require_relative '../config/boot' |
| 8 | 4 |
require 'rails/commands' |
+ 0
- 4
bin/rake
@@ -1,8 +1,4 @@ |
||
| 1 | 1 |
#!/usr/bin/env ruby |
| 2 |
-begin |
|
| 3 |
- load File.expand_path("../spring", __FILE__)
|
|
| 4 |
-rescue LoadError |
|
| 5 |
-end |
|
| 6 | 2 |
require_relative '../config/boot' |
| 7 | 3 |
require 'rake' |
| 8 | 4 |
Rake.application.run |
+ 0
- 4
bin/rspec
@@ -1,7 +1,3 @@ |
||
| 1 | 1 |
#!/usr/bin/env ruby |
| 2 |
-begin |
|
| 3 |
- load File.expand_path("../spring", __FILE__)
|
|
| 4 |
-rescue LoadError |
|
| 5 |
-end |
|
| 6 | 2 |
require 'bundler/setup' |
| 7 | 3 |
load Gem.bin_path('rspec-core', 'rspec')
|
+ 0
- 15
bin/spring
@@ -1,15 +0,0 @@ |
||
| 1 |
-#!/usr/bin/env ruby |
|
| 2 |
- |
|
| 3 |
-# This file loads spring without using Bundler, in order to be fast. |
|
| 4 |
-# It gets overwritten when you run the `spring binstub` command. |
|
| 5 |
- |
|
| 6 |
-unless defined?(Spring) |
|
| 7 |
- require "rubygems" |
|
| 8 |
- require "bundler" |
|
| 9 |
- |
|
| 10 |
- if match = Bundler.default_lockfile.read.match(/^GEM$.*?^ (?: )*spring \((.*?)\)$.*?^$/m) |
|
| 11 |
- Gem.paths = { "GEM_PATH" => Bundler.bundle_path.to_s }
|
|
| 12 |
- gem "spring", match[1] |
|
| 13 |
- require "spring/binstub" |
|
| 14 |
- end |
|
| 15 |
-end |
+ 5
- 2
config/environments/development.rb
@@ -39,8 +39,11 @@ Huginn::Application.configure do |
||
| 39 | 39 |
|
| 40 | 40 |
config.action_mailer.default_url_options = { :host => ENV['DOMAIN'] }
|
| 41 | 41 |
config.action_mailer.asset_host = ENV['DOMAIN'] |
| 42 |
- config.action_mailer.perform_deliveries = ENV['SEND_EMAIL_IN_DEVELOPMENT'] == 'true' |
|
| 43 | 42 |
config.action_mailer.raise_delivery_errors = true |
| 44 |
- config.action_mailer.delivery_method = :smtp |
|
| 43 |
+ if ENV['SEND_EMAIL_IN_DEVELOPMENT'] == 'true' |
|
| 44 |
+ config.action_mailer.delivery_method = :smtp |
|
| 45 |
+ else |
|
| 46 |
+ config.action_mailer.delivery_method = :letter_opener_web |
|
| 47 |
+ end |
|
| 45 | 48 |
# smtp_settings moved to config/initializers/action_mailer.rb |
| 46 | 49 |
end |
+ 2
- 12
config/initializers/action_mailer.rb
@@ -2,15 +2,5 @@ |
||
| 2 | 2 |
|
| 3 | 3 |
smtp_config = YAML::load(ERB.new(File.read(Rails.root.join('config', 'smtp.yml'))).result)
|
| 4 | 4 |
if smtp_config.keys.include? Rails.env |
| 5 |
- Huginn::Application.config.action_mailer.smtp_settings = smtp_config[Rails.env].symbolize_keys |
|
| 6 |
-end |
|
| 7 |
- |
|
| 8 |
-# Huginn::Application.config.action_mailer.smtp_settings = {
|
|
| 9 |
-# address: ENV['SMTP_SERVER'] || 'smtp.gmail.com', |
|
| 10 |
-# port: ENV['SMTP_PORT'] || 587, |
|
| 11 |
-# domain: ENV['SMTP_DOMAIN'], |
|
| 12 |
-# authentication: ENV['SMTP_AUTHENTICATION'] || 'plain', |
|
| 13 |
-# enable_starttls_auto: ENV['SMTP_ENABLE_STARTTLS_AUTO'] == 'true' ? true : false, |
|
| 14 |
-# user_name: ENV['SMTP_USER_NAME'], |
|
| 15 |
-# password: ENV['SMTP_PASSWORD'] |
|
| 16 |
-# } |
|
| 5 |
+ ActionMailer::Base.smtp_settings = smtp_config[Rails.env].symbolize_keys |
|
| 6 |
+end |
+ 1
- 1
config/initializers/delayed_job.rb
@@ -1,6 +1,6 @@ |
||
| 1 | 1 |
Delayed::Worker.destroy_failed_jobs = false |
| 2 | 2 |
Delayed::Worker.max_attempts = 5 |
| 3 |
-Delayed::Worker.max_run_time = (ENV['DELAYED_JOB_MAX_RUNTIME'].presence || 20).to_i.minutes |
|
| 3 |
+Delayed::Worker.max_run_time = (ENV['DELAYED_JOB_MAX_RUNTIME'].presence || 2).to_i.minutes |
|
| 4 | 4 |
Delayed::Worker.read_ahead = 5 |
| 5 | 5 |
Delayed::Worker.default_priority = 10 |
| 6 | 6 |
Delayed::Worker.delay_jobs = !Rails.env.test? |
+ 8
- 0
config/initializers/liquid.rb
@@ -0,0 +1,8 @@ |
||
| 1 |
+module Liquid |
|
| 2 |
+ # https://github.com/Shopify/liquid/pull/623 |
|
| 3 |
+ remove_const :PartialTemplateParser |
|
| 4 |
+ remove_const :TemplateParser |
|
| 5 |
+ |
|
| 6 |
+ PartialTemplateParser = /#{TagStart}.*?#{TagEnd}|#{VariableStart}(?:(?:[^'"{}]+|#{QuotedString})*?|.*?)#{VariableIncompleteEnd}/m
|
|
| 7 |
+ TemplateParser = /(#{PartialTemplateParser}|#{AnyStartingTag})/m
|
|
| 8 |
+end |
+ 6
- 1
config/routes.rb
@@ -2,7 +2,7 @@ Huginn::Application.routes.draw do |
||
| 2 | 2 |
resources :agents do |
| 3 | 3 |
member do |
| 4 | 4 |
post :run |
| 5 |
- put :dry_run |
|
| 5 |
+ post :dry_run |
|
| 6 | 6 |
post :handle_details_post |
| 7 | 7 |
put :leave_scenario |
| 8 | 8 |
delete :remove_events |
@@ -62,6 +62,7 @@ Huginn::Application.routes.draw do |
||
| 62 | 62 |
end |
| 63 | 63 |
collection do |
| 64 | 64 |
delete :destroy_failed |
| 65 |
+ delete :destroy_all |
|
| 65 | 66 |
end |
| 66 | 67 |
end |
| 67 | 68 |
|
@@ -74,6 +75,10 @@ Huginn::Application.routes.draw do |
||
| 74 | 75 |
devise_for :users, |
| 75 | 76 |
controllers: { omniauth_callbacks: 'omniauth_callbacks' },
|
| 76 | 77 |
sign_out_via: [:post, :delete] |
| 78 |
+ |
|
| 79 |
+ if Rails.env.development? |
|
| 80 |
+ mount LetterOpenerWeb::Engine, at: "/letter_opener" |
|
| 81 |
+ end |
|
| 77 | 82 |
|
| 78 | 83 |
get "/about" => "home#about" |
| 79 | 84 |
root :to => "home#index" |
+ 13
- 0
db/migrate/20150507153436_update_keep_events_for_to_be_in_seconds.rb
@@ -0,0 +1,13 @@ |
||
| 1 |
+class UpdateKeepEventsForToBeInSeconds < ActiveRecord::Migration |
|
| 2 |
+ class Agent < ActiveRecord::Base; end |
|
| 3 |
+ |
|
| 4 |
+ SECONDS_IN_DAY = 60 * 60 * 24 |
|
| 5 |
+ |
|
| 6 |
+ def up |
|
| 7 |
+ Agent.update_all ['keep_events_for = keep_events_for * ?', SECONDS_IN_DAY] |
|
| 8 |
+ end |
|
| 9 |
+ |
|
| 10 |
+ def down |
|
| 11 |
+ Agent.update_all ['keep_events_for = keep_events_for / ?', SECONDS_IN_DAY] |
|
| 12 |
+ end |
|
| 13 |
+end |
+ 5
- 0
db/migrate/20150808115436_remove_requirement_from_users_invitation_code.rb
@@ -0,0 +1,5 @@ |
||
| 1 |
+class RemoveRequirementFromUsersInvitationCode < ActiveRecord::Migration |
|
| 2 |
+ def change |
|
| 3 |
+ change_column_null :users, :invitation_code, true, ENV['INVITATION_CODE'].presence || 'try-huginn' |
|
| 4 |
+ end |
|
| 5 |
+end |
+ 77
- 77
db/schema.rb
@@ -11,39 +11,39 @@ |
||
| 11 | 11 |
# |
| 12 | 12 |
# It's strongly recommended that you check this file into your version control system. |
| 13 | 13 |
|
| 14 |
-ActiveRecord::Schema.define(version: 20140906030139) do |
|
| 15 |
- |
|
| 16 |
- create_table "agent_logs", force: true do |t| |
|
| 17 |
- t.integer "agent_id", null: false |
|
| 18 |
- t.text "message", null: false, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 19 |
- t.integer "level", default: 3, null: false |
|
| 20 |
- t.integer "inbound_event_id" |
|
| 21 |
- t.integer "outbound_event_id" |
|
| 14 |
+ActiveRecord::Schema.define(version: 20150808115436) do |
|
| 15 |
+ |
|
| 16 |
+ create_table "agent_logs", force: :cascade do |t| |
|
| 17 |
+ t.integer "agent_id", limit: 4, null: false |
|
| 18 |
+ t.text "message", limit: 65535, null: false, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 19 |
+ t.integer "level", limit: 4, default: 3, null: false |
|
| 20 |
+ t.integer "inbound_event_id", limit: 4 |
|
| 21 |
+ t.integer "outbound_event_id", limit: 4 |
|
| 22 | 22 |
t.datetime "created_at" |
| 23 | 23 |
t.datetime "updated_at" |
| 24 | 24 |
end |
| 25 | 25 |
|
| 26 |
- create_table "agents", force: true do |t| |
|
| 27 |
- t.integer "user_id" |
|
| 28 |
- t.text "options", charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 29 |
- t.string "type", collation: "utf8_bin" |
|
| 30 |
- t.string "name", charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 31 |
- t.string "schedule", collation: "utf8_bin" |
|
| 32 |
- t.integer "events_count", default: 0, null: false |
|
| 26 |
+ create_table "agents", force: :cascade do |t| |
|
| 27 |
+ t.integer "user_id", limit: 4 |
|
| 28 |
+ t.text "options", limit: 65535, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 29 |
+ t.string "type", limit: 255, collation: "utf8_bin" |
|
| 30 |
+ t.string "name", limit: 255, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 31 |
+ t.string "schedule", limit: 255, collation: "utf8_bin" |
|
| 32 |
+ t.integer "events_count", limit: 4, default: 0, null: false |
|
| 33 | 33 |
t.datetime "last_check_at" |
| 34 | 34 |
t.datetime "last_receive_at" |
| 35 |
- t.integer "last_checked_event_id" |
|
| 35 |
+ t.integer "last_checked_event_id", limit: 4 |
|
| 36 | 36 |
t.datetime "created_at" |
| 37 | 37 |
t.datetime "updated_at" |
| 38 |
- t.text "memory", limit: 2147483647, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 38 |
+ t.text "memory", limit: 4294967295, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 39 | 39 |
t.datetime "last_web_request_at" |
| 40 |
- t.integer "keep_events_for", default: 0, null: false |
|
| 40 |
+ t.integer "keep_events_for", limit: 4, default: 0, null: false |
|
| 41 | 41 |
t.datetime "last_event_at" |
| 42 | 42 |
t.datetime "last_error_log_at" |
| 43 |
- t.boolean "propagate_immediately", default: false, null: false |
|
| 44 |
- t.boolean "disabled", default: false, null: false |
|
| 45 |
- t.integer "service_id" |
|
| 46 |
- t.string "guid", null: false |
|
| 43 |
+ t.boolean "propagate_immediately", limit: 1, default: false, null: false |
|
| 44 |
+ t.boolean "disabled", limit: 1, default: false, null: false |
|
| 45 |
+ t.string "guid", limit: 255, null: false, charset: "ascii", collation: "ascii_bin" |
|
| 46 |
+ t.integer "service_id", limit: 4 |
|
| 47 | 47 |
end |
| 48 | 48 |
|
| 49 | 49 |
add_index "agents", ["guid"], name: "index_agents_on_guid", using: :btree |
@@ -51,9 +51,9 @@ ActiveRecord::Schema.define(version: 20140906030139) do |
||
| 51 | 51 |
add_index "agents", ["type"], name: "index_agents_on_type", using: :btree |
| 52 | 52 |
add_index "agents", ["user_id", "created_at"], name: "index_agents_on_user_id_and_created_at", using: :btree |
| 53 | 53 |
|
| 54 |
- create_table "control_links", force: true do |t| |
|
| 55 |
- t.integer "controller_id", null: false |
|
| 56 |
- t.integer "control_target_id", null: false |
|
| 54 |
+ create_table "control_links", force: :cascade do |t| |
|
| 55 |
+ t.integer "controller_id", limit: 4, null: false |
|
| 56 |
+ t.integer "control_target_id", limit: 4, null: false |
|
| 57 | 57 |
t.datetime "created_at" |
| 58 | 58 |
t.datetime "updated_at" |
| 59 | 59 |
end |
@@ -61,25 +61,25 @@ ActiveRecord::Schema.define(version: 20140906030139) do |
||
| 61 | 61 |
add_index "control_links", ["control_target_id"], name: "index_control_links_on_control_target_id", using: :btree |
| 62 | 62 |
add_index "control_links", ["controller_id", "control_target_id"], name: "index_control_links_on_controller_id_and_control_target_id", unique: true, using: :btree |
| 63 | 63 |
|
| 64 |
- create_table "delayed_jobs", force: true do |t| |
|
| 65 |
- t.integer "priority", default: 0 |
|
| 66 |
- t.integer "attempts", default: 0 |
|
| 64 |
+ create_table "delayed_jobs", force: :cascade do |t| |
|
| 65 |
+ t.integer "priority", limit: 4, default: 0 |
|
| 66 |
+ t.integer "attempts", limit: 4, default: 0 |
|
| 67 | 67 |
t.text "handler", limit: 16777215, charset: "utf8mb4", collation: "utf8mb4_bin" |
| 68 |
- t.text "last_error", charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 68 |
+ t.text "last_error", limit: 65535, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 69 | 69 |
t.datetime "run_at" |
| 70 | 70 |
t.datetime "locked_at" |
| 71 | 71 |
t.datetime "failed_at" |
| 72 |
- t.string "locked_by" |
|
| 73 |
- t.string "queue" |
|
| 72 |
+ t.string "locked_by", limit: 255 |
|
| 73 |
+ t.string "queue", limit: 255 |
|
| 74 | 74 |
t.datetime "created_at" |
| 75 | 75 |
t.datetime "updated_at" |
| 76 | 76 |
end |
| 77 | 77 |
|
| 78 | 78 |
add_index "delayed_jobs", ["priority", "run_at"], name: "delayed_jobs_priority", using: :btree |
| 79 | 79 |
|
| 80 |
- create_table "events", force: true do |t| |
|
| 81 |
- t.integer "user_id" |
|
| 82 |
- t.integer "agent_id" |
|
| 80 |
+ create_table "events", force: :cascade do |t| |
|
| 81 |
+ t.integer "user_id", limit: 4 |
|
| 82 |
+ t.integer "agent_id", limit: 4 |
|
| 83 | 83 |
t.decimal "lat", precision: 15, scale: 10 |
| 84 | 84 |
t.decimal "lng", precision: 15, scale: 10 |
| 85 | 85 |
t.text "payload", limit: 16777215, charset: "utf8mb4", collation: "utf8mb4_bin" |
@@ -92,20 +92,20 @@ ActiveRecord::Schema.define(version: 20140906030139) do |
||
| 92 | 92 |
add_index "events", ["expires_at"], name: "index_events_on_expires_at", using: :btree |
| 93 | 93 |
add_index "events", ["user_id", "created_at"], name: "index_events_on_user_id_and_created_at", using: :btree |
| 94 | 94 |
|
| 95 |
- create_table "links", force: true do |t| |
|
| 96 |
- t.integer "source_id" |
|
| 97 |
- t.integer "receiver_id" |
|
| 95 |
+ create_table "links", force: :cascade do |t| |
|
| 96 |
+ t.integer "source_id", limit: 4 |
|
| 97 |
+ t.integer "receiver_id", limit: 4 |
|
| 98 | 98 |
t.datetime "created_at" |
| 99 | 99 |
t.datetime "updated_at" |
| 100 |
- t.integer "event_id_at_creation", default: 0, null: false |
|
| 100 |
+ t.integer "event_id_at_creation", limit: 4, default: 0, null: false |
|
| 101 | 101 |
end |
| 102 | 102 |
|
| 103 | 103 |
add_index "links", ["receiver_id", "source_id"], name: "index_links_on_receiver_id_and_source_id", using: :btree |
| 104 | 104 |
add_index "links", ["source_id", "receiver_id"], name: "index_links_on_source_id_and_receiver_id", using: :btree |
| 105 | 105 |
|
| 106 |
- create_table "scenario_memberships", force: true do |t| |
|
| 107 |
- t.integer "agent_id", null: false |
|
| 108 |
- t.integer "scenario_id", null: false |
|
| 106 |
+ create_table "scenario_memberships", force: :cascade do |t| |
|
| 107 |
+ t.integer "agent_id", limit: 4, null: false |
|
| 108 |
+ t.integer "scenario_id", limit: 4, null: false |
|
| 109 | 109 |
t.datetime "created_at" |
| 110 | 110 |
t.datetime "updated_at" |
| 111 | 111 |
end |
@@ -113,71 +113,71 @@ ActiveRecord::Schema.define(version: 20140906030139) do |
||
| 113 | 113 |
add_index "scenario_memberships", ["agent_id"], name: "index_scenario_memberships_on_agent_id", using: :btree |
| 114 | 114 |
add_index "scenario_memberships", ["scenario_id"], name: "index_scenario_memberships_on_scenario_id", using: :btree |
| 115 | 115 |
|
| 116 |
- create_table "scenarios", force: true do |t| |
|
| 117 |
- t.string "name", null: false, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 118 |
- t.integer "user_id", null: false |
|
| 116 |
+ create_table "scenarios", force: :cascade do |t| |
|
| 117 |
+ t.string "name", limit: 255, null: false, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 118 |
+ t.integer "user_id", limit: 4, null: false |
|
| 119 | 119 |
t.datetime "created_at" |
| 120 | 120 |
t.datetime "updated_at" |
| 121 |
- t.text "description", charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 122 |
- t.boolean "public", default: false, null: false |
|
| 123 |
- t.string "guid", null: false, charset: "ascii", collation: "ascii_bin" |
|
| 124 |
- t.string "source_url" |
|
| 125 |
- t.string "tag_bg_color" |
|
| 126 |
- t.string "tag_fg_color" |
|
| 121 |
+ t.text "description", limit: 65535, charset: "utf8mb4", collation: "utf8mb4_bin" |
|
| 122 |
+ t.boolean "public", limit: 1, default: false, null: false |
|
| 123 |
+ t.string "guid", limit: 255, null: false, charset: "ascii", collation: "ascii_bin" |
|
| 124 |
+ t.string "source_url", limit: 255 |
|
| 125 |
+ t.string "tag_bg_color", limit: 255 |
|
| 126 |
+ t.string "tag_fg_color", limit: 255 |
|
| 127 | 127 |
end |
| 128 | 128 |
|
| 129 | 129 |
add_index "scenarios", ["user_id", "guid"], name: "index_scenarios_on_user_id_and_guid", unique: true, using: :btree |
| 130 | 130 |
|
| 131 |
- create_table "services", force: true do |t| |
|
| 132 |
- t.integer "user_id", null: false |
|
| 133 |
- t.string "provider", null: false |
|
| 134 |
- t.string "name", null: false |
|
| 135 |
- t.text "token", null: false |
|
| 136 |
- t.text "secret" |
|
| 137 |
- t.text "refresh_token" |
|
| 131 |
+ create_table "services", force: :cascade do |t| |
|
| 132 |
+ t.integer "user_id", limit: 4, null: false |
|
| 133 |
+ t.string "provider", limit: 255, null: false, collation: "utf8_general_ci" |
|
| 134 |
+ t.string "name", limit: 255, null: false, collation: "utf8_general_ci" |
|
| 135 |
+ t.text "token", limit: 65535, null: false, collation: "utf8_general_ci" |
|
| 136 |
+ t.text "secret", limit: 65535, collation: "utf8_general_ci" |
|
| 137 |
+ t.text "refresh_token", limit: 65535, collation: "utf8_general_ci" |
|
| 138 | 138 |
t.datetime "expires_at" |
| 139 |
- t.boolean "global", default: false |
|
| 140 |
- t.text "options" |
|
| 139 |
+ t.boolean "global", limit: 1, default: false |
|
| 140 |
+ t.text "options", limit: 65535, collation: "utf8_general_ci" |
|
| 141 | 141 |
t.datetime "created_at" |
| 142 | 142 |
t.datetime "updated_at" |
| 143 |
- t.string "uid" |
|
| 143 |
+ t.string "uid", limit: 255, collation: "utf8_general_ci" |
|
| 144 | 144 |
end |
| 145 | 145 |
|
| 146 | 146 |
add_index "services", ["provider"], name: "index_services_on_provider", using: :btree |
| 147 | 147 |
add_index "services", ["uid"], name: "index_services_on_uid", using: :btree |
| 148 | 148 |
add_index "services", ["user_id", "global"], name: "index_services_on_user_id_and_global", using: :btree |
| 149 | 149 |
|
| 150 |
- create_table "user_credentials", force: true do |t| |
|
| 151 |
- t.integer "user_id", null: false |
|
| 152 |
- t.string "credential_name", null: false |
|
| 153 |
- t.text "credential_value", null: false |
|
| 150 |
+ create_table "user_credentials", force: :cascade do |t| |
|
| 151 |
+ t.integer "user_id", limit: 4, null: false |
|
| 152 |
+ t.string "credential_name", limit: 255, null: false |
|
| 153 |
+ t.text "credential_value", limit: 65535, null: false |
|
| 154 | 154 |
t.datetime "created_at" |
| 155 | 155 |
t.datetime "updated_at" |
| 156 |
- t.string "mode", default: "text", null: false, collation: "utf8_bin" |
|
| 156 |
+ t.string "mode", limit: 255, default: "text", null: false, collation: "utf8_bin" |
|
| 157 | 157 |
end |
| 158 | 158 |
|
| 159 | 159 |
add_index "user_credentials", ["user_id", "credential_name"], name: "index_user_credentials_on_user_id_and_credential_name", unique: true, using: :btree |
| 160 | 160 |
|
| 161 |
- create_table "users", force: true do |t| |
|
| 162 |
- t.string "email", default: "", null: false, collation: "utf8_bin" |
|
| 163 |
- t.string "encrypted_password", default: "", null: false, charset: "ascii", collation: "ascii_bin" |
|
| 164 |
- t.string "reset_password_token", collation: "utf8_bin" |
|
| 161 |
+ create_table "users", force: :cascade do |t| |
|
| 162 |
+ t.string "email", limit: 255, default: "", null: false, collation: "utf8_bin" |
|
| 163 |
+ t.string "encrypted_password", limit: 255, default: "", null: false, charset: "ascii", collation: "ascii_bin" |
|
| 164 |
+ t.string "reset_password_token", limit: 255, collation: "utf8_bin" |
|
| 165 | 165 |
t.datetime "reset_password_sent_at" |
| 166 | 166 |
t.datetime "remember_created_at" |
| 167 |
- t.integer "sign_in_count", default: 0 |
|
| 167 |
+ t.integer "sign_in_count", limit: 4, default: 0 |
|
| 168 | 168 |
t.datetime "current_sign_in_at" |
| 169 | 169 |
t.datetime "last_sign_in_at" |
| 170 |
- t.string "current_sign_in_ip" |
|
| 171 |
- t.string "last_sign_in_ip" |
|
| 170 |
+ t.string "current_sign_in_ip", limit: 255 |
|
| 171 |
+ t.string "last_sign_in_ip", limit: 255 |
|
| 172 | 172 |
t.datetime "created_at" |
| 173 | 173 |
t.datetime "updated_at" |
| 174 |
- t.boolean "admin", default: false, null: false |
|
| 175 |
- t.integer "failed_attempts", default: 0 |
|
| 176 |
- t.string "unlock_token" |
|
| 174 |
+ t.boolean "admin", limit: 1, default: false, null: false |
|
| 175 |
+ t.integer "failed_attempts", limit: 4, default: 0 |
|
| 176 |
+ t.string "unlock_token", limit: 255 |
|
| 177 | 177 |
t.datetime "locked_at" |
| 178 | 178 |
t.string "username", limit: 191, null: false, charset: "utf8mb4", collation: "utf8mb4_unicode_ci" |
| 179 |
- t.string "invitation_code", null: false, collation: "utf8_bin" |
|
| 180 |
- t.integer "scenario_count", default: 0, null: false |
|
| 179 |
+ t.string "invitation_code", limit: 255, collation: "utf8_bin" |
|
| 180 |
+ t.integer "scenario_count", limit: 4, default: 0, null: false |
|
| 181 | 181 |
end |
| 182 | 182 |
|
| 183 | 183 |
add_index "users", ["email"], name: "index_users_on_email", unique: true, using: :btree |
+ 0
- 5
deployment/heroku/unicorn.rb
@@ -17,11 +17,6 @@ Thread.new do |
||
| 17 | 17 |
|
| 18 | 18 |
sleep 45 |
| 19 | 19 |
|
| 20 |
- if ENV['DOMAIN'] |
|
| 21 |
- force_ssl = ENV['FORCE_SSL'] == 'true' |
|
| 22 |
- Net::HTTP.get_response(URI((force_ssl ? "https://" : "http://") + ENV['DOMAIN'])) |
|
| 23 |
- end |
|
| 24 |
- |
|
| 25 | 20 |
begin |
| 26 | 21 |
Process.getpgid worker_pid |
| 27 | 22 |
rescue Errno::ESRCH |
+ 6
- 2
deployment/site-cookbooks/huginn_production/files/default/env.example
@@ -39,6 +39,9 @@ FORCE_SSL=false |
||
| 39 | 39 |
# You can see its use in user.rb. PLEASE CHANGE THIS! |
| 40 | 40 |
INVITATION_CODE=try-huginn |
| 41 | 41 |
|
| 42 |
+# If you don't want to require users to have an invitation code, set this to true |
|
| 43 |
+SKIP_INVITATION_CODE=false |
|
| 44 |
+ |
|
| 42 | 45 |
############################# |
| 43 | 46 |
# Email Configuration # |
| 44 | 47 |
############################# |
@@ -46,7 +49,7 @@ INVITATION_CODE=try-huginn |
||
| 46 | 49 |
# Outgoing email settings. To use Gmail or Google Apps, put your Google Apps domain or gmail.com |
| 47 | 50 |
# as the SMTP_DOMAIN and your Gmail username and password as the SMTP_USER_NAME and SMTP_PASSWORD. |
| 48 | 51 |
# |
| 49 |
-# PLEASE NOTE: In order to enable emails locally (e.g., when not in the production Rails environment), |
|
| 52 |
+# PLEASE NOTE: In order to enable sending real emails via SMTP locally (e.g., when not in the production Rails environment), |
|
| 50 | 53 |
# you must also set SEND_EMAIL_IN_DEVELOPMENT to true below. |
| 51 | 54 |
|
| 52 | 55 |
SMTP_DOMAIN=your-domain-here.com |
@@ -57,7 +60,8 @@ SMTP_PORT=587 |
||
| 57 | 60 |
SMTP_AUTHENTICATION=plain |
| 58 | 61 |
SMTP_ENABLE_STARTTLS_AUTO=true |
| 59 | 62 |
|
| 60 |
-# Send emails when running in the development Rails environment. |
|
| 63 |
+# Set to true to send real emails via SMTP when running in the development Rails environment. |
|
| 64 |
+# Set to false to have emails intercepted in development and displayed at http://localhost:3000/letter_opener |
|
| 61 | 65 |
SEND_EMAIL_IN_DEVELOPMENT=false |
| 62 | 66 |
|
| 63 | 67 |
# The address from which system emails will appear to be sent. |
+ 7
- 2
docker/scripts/init
@@ -6,6 +6,11 @@ cd /app |
||
| 6 | 6 |
# Default to the environment variable values set in .env.example |
| 7 | 7 |
source /app/.env.example |
| 8 | 8 |
|
| 9 |
+# Cleanup any leftover pid file |
|
| 10 |
+if [ -f /app/tmp/pids/server.pid ]; then |
|
| 11 |
+ rm /app/tmp/pids/server.pid |
|
| 12 |
+fi |
|
| 13 |
+ |
|
| 9 | 14 |
# is a mysql or postgresql database linked? |
| 10 | 15 |
# requires that the mysql or postgresql containers have exposed |
| 11 | 16 |
# port 3306 and 5432 respectively. |
@@ -150,8 +155,8 @@ source /app/.env |
||
| 150 | 155 |
|
| 151 | 156 |
# Fixup the Procfile and prepare the PORT |
| 152 | 157 |
[ -n "\${DO_NOT_RUN_JOBS}" ] && perl -pi -e 's/^jobs:/#jobs:/' /app/Procfile
|
| 153 |
-perl -pi -e 's/rails server\$/rails server -b 0.0.0.0 -p \\\$PORT/' /app/Procfile |
|
| 154 |
-export PORT |
|
| 158 |
+perl -pi -e 's/rails server -b0.0.0.0\$/rails server -b 0.0.0.0 -p \\\$PORT/' /app/Procfile |
|
| 159 |
+export PORT="\${PORT:-3000}"
|
|
| 155 | 160 |
|
| 156 | 161 |
# Start huginn |
| 157 | 162 |
exec sudo -u huginn -EH bundle exec foreman start |
+ 17
- 3
lib/agents_exporter.rb
@@ -12,6 +12,7 @@ class AgentsExporter |
||
| 12 | 12 |
|
| 13 | 13 |
def as_json(opts = {})
|
| 14 | 14 |
{
|
| 15 |
+ :schema_version => 1, |
|
| 15 | 16 |
:name => options[:name].presence || 'No name provided', |
| 16 | 17 |
:description => options[:description].presence || 'No description provided', |
| 17 | 18 |
:source_url => options[:source_url], |
@@ -20,7 +21,8 @@ class AgentsExporter |
||
| 20 | 21 |
:tag_bg_color => options[:tag_bg_color], |
| 21 | 22 |
:exported_at => Time.now.utc.iso8601, |
| 22 | 23 |
:agents => agents.map { |agent| agent_as_json(agent) },
|
| 23 |
- :links => links |
|
| 24 |
+ :links => links, |
|
| 25 |
+ :control_links => control_links |
|
| 24 | 26 |
} |
| 25 | 27 |
end |
| 26 | 28 |
|
@@ -32,14 +34,26 @@ class AgentsExporter |
||
| 32 | 34 |
agent_ids = agents.map(&:id) |
| 33 | 35 |
|
| 34 | 36 |
contained_links = agents.map.with_index do |agent, index| |
| 35 |
- agent.links_as_source.where(:receiver_id => agent_ids).map do |link| |
|
| 36 |
- { :source => index, :receiver => agent_ids.index(link.receiver_id) }
|
|
| 37 |
+ agent.links_as_source.where(receiver_id: agent_ids).map do |link| |
|
| 38 |
+ { source: index, receiver: agent_ids.index(link.receiver_id) }
|
|
| 37 | 39 |
end |
| 38 | 40 |
end |
| 39 | 41 |
|
| 40 | 42 |
contained_links.flatten.compact |
| 41 | 43 |
end |
| 42 | 44 |
|
| 45 |
+ def control_links |
|
| 46 |
+ agent_ids = agents.map(&:id) |
|
| 47 |
+ |
|
| 48 |
+ contained_controller_links = agents.map.with_index do |agent, index| |
|
| 49 |
+ agent.control_links_as_controller.where(control_target_id: agent_ids).map do |control_link| |
|
| 50 |
+ { controller: index, control_target: agent_ids.index(control_link.control_target_id) }
|
|
| 51 |
+ end |
|
| 52 |
+ end |
|
| 53 |
+ |
|
| 54 |
+ contained_controller_links.flatten.compact |
|
| 55 |
+ end |
|
| 56 |
+ |
|
| 43 | 57 |
def agent_as_json(agent) |
| 44 | 58 |
{
|
| 45 | 59 |
:type => agent.type, |
+ 0
- 3
lib/ar_mysql_column_charset.rb
@@ -1,6 +1,3 @@ |
||
| 1 |
-# Module#prepend support for Ruby 1.9 |
|
| 2 |
-require 'prepend' unless Module.method_defined?(:prepend) |
|
| 3 |
- |
|
| 4 | 1 |
require 'active_support' |
| 5 | 2 |
|
| 6 | 3 |
ActiveSupport.on_load :active_record do |
+ 1
- 1
lib/huginn_scheduler.rb
@@ -114,7 +114,7 @@ class HuginnScheduler |
||
| 114 | 114 |
end |
| 115 | 115 |
|
| 116 | 116 |
# Schedule event cleanup. |
| 117 |
- @rufus_scheduler.cron "0 0 * * * " + tzinfo_friendly_timezone do |
|
| 117 |
+ @rufus_scheduler.every ENV['EVENT_EXPIRATION_CHECK'].presence || '6h' do |
|
| 118 | 118 |
cleanup_expired_events! |
| 119 | 119 |
end |
| 120 | 120 |
|
+ 0
- 85
lib/prepend.rb
@@ -1,85 +0,0 @@ |
||
| 1 |
-# Fake implementation of prepend(), which does not support overriding |
|
| 2 |
-# inherited methods nor methods that are formerly overridden by |
|
| 3 |
-# another invocation of prepend(). |
|
| 4 |
-# |
|
| 5 |
-# Here's what <Original>.prepend(<Wrapper>) does: |
|
| 6 |
-# |
|
| 7 |
-# - Create an anonymous stub module (hereinafter <Stub>) and define |
|
| 8 |
-# <Stub>#<method> that calls #<method>_without_<Wrapper> for each |
|
| 9 |
-# instance method of <Wrapper>. |
|
| 10 |
-# |
|
| 11 |
-# - Rename <Original>#<method> to #<method>_without_<Wrapper> for each |
|
| 12 |
-# instance method of <Wrapper>. |
|
| 13 |
-# |
|
| 14 |
-# - Include <Stub> and <Wrapper> into <Original> in that order. |
|
| 15 |
-# |
|
| 16 |
-# This way, a call of <Original>#<method> is dispatched to |
|
| 17 |
-# <Wrapper><method>, which may call super which is dispatched to |
|
| 18 |
-# <Stub>#<method>, which finally calls |
|
| 19 |
-# <Original>#<method>_without_<Wrapper> which is used to be called |
|
| 20 |
-# <Original>#<method>. |
|
| 21 |
-# |
|
| 22 |
-# Usage: |
|
| 23 |
-# |
|
| 24 |
-# class Mechanize |
|
| 25 |
-# # module with methods that overrides those of X |
|
| 26 |
-# module Y |
|
| 27 |
-# end |
|
| 28 |
-# |
|
| 29 |
-# unless X.respond_to?(:prepend, true) |
|
| 30 |
-# require 'mechanize/prependable' |
|
| 31 |
-# X.extend(Prependable) |
|
| 32 |
-# end |
|
| 33 |
-# |
|
| 34 |
-# class X |
|
| 35 |
-# prepend Y |
|
| 36 |
-# end |
|
| 37 |
-# end |
|
| 38 |
-class Module |
|
| 39 |
- def prepend(mod) |
|
| 40 |
- stub = Module.new |
|
| 41 |
- |
|
| 42 |
- mod_id = (mod.name || 'Module__%d' % mod.object_id).gsub(/::/, '__') |
|
| 43 |
- |
|
| 44 |
- mod.instance_methods.each { |name|
|
|
| 45 |
- method_defined?(name) or next |
|
| 46 |
- |
|
| 47 |
- original = instance_method(name) |
|
| 48 |
- next if original.owner != self |
|
| 49 |
- |
|
| 50 |
- name = name.to_s |
|
| 51 |
- name_without = name.sub(/(?=[?!=]?\z)/) { '_without_%s' % mod_id }
|
|
| 52 |
- |
|
| 53 |
- arity = original.arity |
|
| 54 |
- arglist = ( |
|
| 55 |
- if arity >= 0 |
|
| 56 |
- (1..arity).map { |i| 'x%d' % i }
|
|
| 57 |
- else |
|
| 58 |
- (1..(-arity - 1)).map { |i| 'x%d' % i } << '*a'
|
|
| 59 |
- end << '&b' |
|
| 60 |
- ).join(', ')
|
|
| 61 |
- |
|
| 62 |
- if name.end_with?('=')
|
|
| 63 |
- stub.module_eval %{
|
|
| 64 |
- def #{name}(#{arglist})
|
|
| 65 |
- __send__(:#{name_without}, #{arglist})
|
|
| 66 |
- end |
|
| 67 |
- } |
|
| 68 |
- else |
|
| 69 |
- stub.module_eval %{
|
|
| 70 |
- def #{name}(#{arglist})
|
|
| 71 |
- #{name_without}(#{arglist})
|
|
| 72 |
- end |
|
| 73 |
- } |
|
| 74 |
- end |
|
| 75 |
- module_eval {
|
|
| 76 |
- alias_method name_without, name |
|
| 77 |
- remove_method name |
|
| 78 |
- } |
|
| 79 |
- } |
|
| 80 |
- |
|
| 81 |
- include stub |
|
| 82 |
- include mod |
|
| 83 |
- end |
|
| 84 |
- private :prepend |
|
| 85 |
-end unless Module.method_defined?(:prepend) |
+ 105
- 0
lib/tasks/icon.rake
@@ -0,0 +1,105 @@ |
||
| 1 |
+ICONS_DIR = 'public' |
|
| 2 |
+ORIGINAL_IMAGE = 'media/huginn-icon-square.svg' |
|
| 3 |
+ |
|
| 4 |
+desc "Generate site icons from #{ORIGINAL_IMAGE}"
|
|
| 5 |
+task :icons => 'icon:all' |
|
| 6 |
+ |
|
| 7 |
+namespace :icon do |
|
| 8 |
+ # iOS |
|
| 9 |
+ task :all => :ios |
|
| 10 |
+ |
|
| 11 |
+ [ |
|
| 12 |
+ 57, 114, |
|
| 13 |
+ 60, 120, 180, |
|
| 14 |
+ 72, 144, |
|
| 15 |
+ 76, 152, |
|
| 16 |
+ ].each do |width| |
|
| 17 |
+ sizes = '%1$dx%1$d' % width |
|
| 18 |
+ filename = "apple-touch-icon-#{sizes}.png"
|
|
| 19 |
+ icon = File.join(ICONS_DIR, filename) |
|
| 20 |
+ |
|
| 21 |
+ file icon => ORIGINAL_IMAGE do |t| |
|
| 22 |
+ puts "Generating #{t.name}"
|
|
| 23 |
+ convert_image t.source, t.name, width: width |
|
| 24 |
+ end |
|
| 25 |
+ |
|
| 26 |
+ task :ios => icon |
|
| 27 |
+ end |
|
| 28 |
+ |
|
| 29 |
+ # Android |
|
| 30 |
+ task :all => :android |
|
| 31 |
+ |
|
| 32 |
+ android_icons = [ |
|
| 33 |
+ 36, 72, 144, |
|
| 34 |
+ 48, 96, 192, |
|
| 35 |
+ ].map do |width| |
|
| 36 |
+ sizes = '%1$dx%1$d' % width |
|
| 37 |
+ filename = "android-chrome-#{sizes}.png" % width
|
|
| 38 |
+ icon = File.join(ICONS_DIR, filename) |
|
| 39 |
+ |
|
| 40 |
+ file icon => ORIGINAL_IMAGE do |t| |
|
| 41 |
+ puts "Generating #{t.name}"
|
|
| 42 |
+ convert_image t.source, t.name, width: width, round: true |
|
| 43 |
+ end |
|
| 44 |
+ |
|
| 45 |
+ task :android => icon |
|
| 46 |
+ |
|
| 47 |
+ {
|
|
| 48 |
+ src: "/#{filename}",
|
|
| 49 |
+ sizes: sizes, |
|
| 50 |
+ type: 'image/png', |
|
| 51 |
+ density: (width / 48.0).to_s, |
|
| 52 |
+ } |
|
| 53 |
+ end |
|
| 54 |
+ |
|
| 55 |
+ manifest = File.join(ICONS_DIR, 'manifest.json') |
|
| 56 |
+ |
|
| 57 |
+ file manifest => __FILE__ do |t| |
|
| 58 |
+ puts "Generating #{t.name}"
|
|
| 59 |
+ require 'json' |
|
| 60 |
+ json = {
|
|
| 61 |
+ name: 'Huginn', |
|
| 62 |
+ icons: android_icons |
|
| 63 |
+ } |
|
| 64 |
+ File.write(t.name, JSON.pretty_generate(json)) |
|
| 65 |
+ end |
|
| 66 |
+ |
|
| 67 |
+ task :android => manifest |
|
| 68 |
+end |
|
| 69 |
+ |
|
| 70 |
+require 'mini_magick' |
|
| 71 |
+ |
|
| 72 |
+def convert_image(source, target, options = {}) # width: nil, round: false
|
|
| 73 |
+ ext = target[/(?<=\.)[^.]+\z/] || 'png' |
|
| 74 |
+ original = MiniMagick::Image.open(source) |
|
| 75 |
+ |
|
| 76 |
+ result = original |
|
| 77 |
+ result.format ext |
|
| 78 |
+ |
|
| 79 |
+ if width = options[:width] |
|
| 80 |
+ result.thumbnail '%1$dx%1$d>' % width |
|
| 81 |
+ else |
|
| 82 |
+ width = result[:width] |
|
| 83 |
+ end |
|
| 84 |
+ |
|
| 85 |
+ if options[:round] |
|
| 86 |
+ radius = (Rational(80, 512) * width).round |
|
| 87 |
+ |
|
| 88 |
+ mask = MiniMagick::Image.create(ext) { |tmp| result.write(tmp) }
|
|
| 89 |
+ |
|
| 90 |
+ mask.mogrify do |image| |
|
| 91 |
+ image.alpha 'transparent' |
|
| 92 |
+ image.background 'none' |
|
| 93 |
+ image.fill 'white' |
|
| 94 |
+ image.draw 'roundrectangle 0,0,%1$d,%1$d,%2$d,%2$d' % [width, radius] |
|
| 95 |
+ end |
|
| 96 |
+ |
|
| 97 |
+ result = result.composite(mask) do |image| |
|
| 98 |
+ image.alpha 'set' |
|
| 99 |
+ image.compose 'DstIn' |
|
| 100 |
+ end |
|
| 101 |
+ end |
|
| 102 |
+ |
|
| 103 |
+ result.strip |
|
| 104 |
+ result.write(target) |
|
| 105 |
+end |
+ 39
- 0
lib/utils.rb
@@ -79,4 +79,43 @@ module Utils |
||
| 79 | 79 |
def self.pretty_jsonify(thing) |
| 80 | 80 |
JSON.pretty_generate(thing).gsub('</', '<\/')
|
| 81 | 81 |
end |
| 82 |
+ |
|
| 83 |
+ class TupleSorter |
|
| 84 |
+ class SortableTuple |
|
| 85 |
+ attr_reader :array |
|
| 86 |
+ |
|
| 87 |
+ # The <=> method will call orders[n] to determine if the nth element |
|
| 88 |
+ # should be compared in descending order. |
|
| 89 |
+ def initialize(array, orders = []) |
|
| 90 |
+ @array = array |
|
| 91 |
+ @orders = orders |
|
| 92 |
+ end |
|
| 93 |
+ |
|
| 94 |
+ def <=> other |
|
| 95 |
+ other = other.array |
|
| 96 |
+ @array.each_with_index do |e, i| |
|
| 97 |
+ o = other[i] |
|
| 98 |
+ case cmp = e <=> o || e.to_s <=> o.to_s |
|
| 99 |
+ when 0 |
|
| 100 |
+ next |
|
| 101 |
+ else |
|
| 102 |
+ return @orders[i] ? -cmp : cmp |
|
| 103 |
+ end |
|
| 104 |
+ end |
|
| 105 |
+ 0 |
|
| 106 |
+ end |
|
| 107 |
+ end |
|
| 108 |
+ |
|
| 109 |
+ class << self |
|
| 110 |
+ def sort!(array, orders = []) |
|
| 111 |
+ array.sort_by! do |e| |
|
| 112 |
+ SortableTuple.new(e, orders) |
|
| 113 |
+ end |
|
| 114 |
+ end |
|
| 115 |
+ end |
|
| 116 |
+ end |
|
| 117 |
+ |
|
| 118 |
+ def self.sort_tuples!(array, orders = []) |
|
| 119 |
+ TupleSorter.sort!(array, orders) |
|
| 120 |
+ end |
|
| 82 | 121 |
end |
+ 20
- 0
media/huginn-icon-square.svg
@@ -0,0 +1,20 @@ |
||
| 1 |
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"> |
|
| 2 |
+ <rect fill="#ffb74d" x="0" y="0" width="512" height="512"/> |
|
| 3 |
+ <g id="raven"> |
|
| 4 |
+ <path fill="#212121" d="M 156.25,129.531 C 103.953,130.174 45.819,149.639 37.406,152.875 49.875,151.984 61.227,154.012 64.062,158.656 58.362,161.506 23.505,180.972 0,195.875 0.286,299.561 0,406.962 0,512 l 189.906,0 c 22.425,-73.077 32.383,-153.639 93.625,-190.843 26.765,-14.170 108.939,-57.052 196.312,-46.25 0,0 0.203,-16.701 -23.843,-38.968 -24.048,-22.268 -65.775,-33.365 -91.125,-38.562 -25.350,-5.195 -73.373,-3.786 -77.625,-11.5 -4.250,-7.715 -6.124,-18.240 -18.406,-27.843 -12.280,-9.604 -53.998,-27.869 -107.531,-28.5 -1.672,-0.019 -3.375,-0.020 -5.062,0 z"/> |
|
| 5 |
+ <path fill="#fafafa" d="m 226.083,194.751 c -3.826,0 -6.926,3.140 -6.926,6.968 0,3.828 3.099,6.927 6.926,6.927 3.828,0 6.927,-3.099 6.927,-6.927 0,-3.828 -3.099,-6.968 -6.927,-6.968 z"/> |
|
| 6 |
+ <path fill="#ffc107" d="m 214.872,192.955 c -7.783,2.176 -13.623,9.021 -13.623,17.500 0,10.206 8.318,18.468 18.525,18.468 10.206,0 18.468,-8.261 18.468,-18.468 0,-7.736 -4.768,-14.346 -11.514,-17.101 0.827,0.506 1.710,0.913 2.394,1.595 1.102,1.102 1.948,2.416 2.565,3.875 0.616,1.458 0.968,3.104 0.968,4.788 0,1.682 -0.351,3.272 -0.968,4.730 -0.616,1.458 -1.460,2.772 -2.565,3.875 -1.102,1.102 -2.416,2.004 -3.874,2.621 -1.458,0.616 -3.104,0.968 -4.788,0.968 -1.682,0 -3.272,-0.351 -4.730,-0.968 -1.458,-0.616 -2.772,-1.518 -3.875,-2.621 -1.102,-1.102 -2.004,-2.416 -2.621,-3.875 -0.616,-1.458 -0.912,-3.047 -0.912,-4.730 0,-1.682 0.295,-3.328 0.912,-4.788 0.616,-1.458 1.518,-2.772 2.621,-3.875 0.858,-0.858 1.937,-1.422 3.020,-1.994 z"/> |
|
| 7 |
+ <path fill="#bdbdbd" d="m 271.918,173.168 c 8.272,11.402 5.676,19.982 14.761,23.582 8.794,3.227 53.244,1.965 98.446,14.435 45.200,12.468 67.159,32.076 78.610,47.473 -42.384,-5.059 -130.871,1.773 -175.850,16.026 -31.697,10.044 -49.611,19.253 -51.257,12.117 -5.466,-23.693 22.977,-69.173 10.809,-88.180 14.316,6.047 24.179,-9.468 24.477,-25.454 z"/> |
|
| 8 |
+ <path fill="#bdbdbd" d="m 228.687,269.965 c 0,0 -7.242,15.429 -1.889,23.932 5.353,8.501 12.595,8.501 49.438,-4.092 36.843,-12.595 101.712,-25.506 167.211,-24.876 -99.823,10.076 -161.857,30.544 -188.940,40.622 -27.080,10.076 -36.212,3.463 -39.991,-6.927 -3.778,-10.391 1.574,-21.727 14.170,-28.655 z"/> |
|
| 9 |
+ <path fill="#424242" d="M 155.25 141.125 C 133.412 141.257 108.248 143.215 79.75 147.781 C 105.256 152.189 109.656 160.687 109.656 160.687 C 109.656 160.687 59.581 170.129 16.125 198.156 C 30.925 197.527 36.593 204.156 36.593 204.156 C 23.955 212.371 11.699 220.728 0 229.343 L 0 486.625 C 15.582 469.979 31.312 461.406 31.312 461.406 C 26.293 481.306 24.663 497.555 25.093 512 L 70.218 512 C 79.847 503.782 87.406 499.031 87.406 499.031 C 87.406 499.031 88.399 503.689 89.031 512 L 142.593 512 C 139.765 487.092 134.201 461.199 140.812 420.812 C 140.812 420.812 121.495 463.341 104.437 473.156 C 95.234 427.828 104.729 324.044 221.75 243.281 C 179.247 251.538 143.472 211.666 90.156 228.593 C 142.482 198.533 167.197 185.448 203.937 183 C 240.676 180.551 247.870 192.225 256.281 188.281 C 264.283 184.529 265.411 173.708 261.906 167.156 C 256.300 156.674 220.763 140.728 155.25 141.125 z "/> |
|
| 10 |
+ <path fill="#616161" d="M 198.5 158.593 C 154.091 158.654 93.091 169.031 48.937 195.968 C 67.487 195.677 71.25 202.531 71.25 202.531 C 71.25 202.531 32.127 229.043 0 256.593 L 0 273.656 C 9.199 272.762 15.343 273.781 15.343 273.781 C 9.924 278.009 4.830 282.217 0 286.375 L 0 471.656 C 26.800 445.475 60.093 433.156 60.093 433.156 C 52.955 454.899 46.850 485.933 49.031 512 L 55.156 512 C 62.260 503.313 70.207 497.614 84.875 487.437 C 63.761 386.082 147.281 276.925 199 250.375 C 168.089 253.135 108.959 201.101 47 266.5 C 76.489 222.848 123.815 190.074 178.468 177.25 C 229.927 165.176 249.885 188.401 255.656 176.218 C 260.693 165.583 235.295 158.543 198.5 158.593 z "/> |
|
| 11 |
+ <path fill="#b71c1c" d="M 221.760,243.290 C 104.741,324.053 95.241,427.802 104.446,473.131 c 17.058,-9.813 36.367,-52.329 36.367,-52.329 7.713,-51.405 9.369,-109.749 80.945,-177.510 z"/> |
|
| 12 |
+ </g> |
|
| 13 |
+ <g id="headset"> |
|
| 14 |
+ <path fill="#898989" d="m 109.413,183.151 c -36.892,0 -66.805,29.913 -66.805,66.805 0,36.892 29.913,66.805 66.805,66.805 36.892,0 66.807,-29.913 66.807,-66.805 0,-36.892 -29.914,-66.805 -66.807,-66.805 z"/> |
|
| 15 |
+ <path fill="#4c4c4c" d="m 127.565,235.769 c -34.167,3.290 -29.167,33.215 -29.167,33.215 0,0 66.060,67.601 236.221,61.548 7.209,-13.315 2.169,-11.475 2.169,-11.475 0,0 -139.721,3.076 -209.223,-83.288 z"/> |
|
| 16 |
+ <path fill="#b5b5b5" d="m 337.289,310.462 c -8.197,0 -14.855,6.657 -14.855,14.855 0,8.195 6.657,14.812 14.855,14.812 8.195,0 14.812,-6.616 14.812,-14.812 0,-8.197 -6.616,-14.855 -14.812,-14.855 z"/> |
|
| 17 |
+ <path fill="#303030" d="m 109.413,213.403 c -20.196,0 -36.553,16.355 -36.553,36.553 0,20.197 16.357,36.594 36.553,36.594 20.197,0 36.553,-16.397 36.553,-36.594 0,-20.197 -16.355,-36.553 -36.553,-36.553 z"/> |
|
| 18 |
+ <path fill="#303030" d="m 59.215,123.356 c -15.112,0 -27.373,12.260 -27.373,27.373 0,4.449 1.077,8.595 2.961,12.308 l 0.292,0.543 c 0.026,0.050 0.056,0.156 0.084,0.208 l 49.822,98.560 48.822,-24.744 -50.157,-99.270 -0.041,0 C 79.108,129.441 69.874,123.356 59.215,123.356 z"/> |
|
| 19 |
+ </g> |
|
| 20 |
+</svg> |
BIN
media/iOS/Icon-40.png
BIN
media/iOS/Icon-40@2x.png
BIN
media/iOS/Icon-60@2x.png
BIN
media/iOS/Icon-60@3x.png
BIN
media/iOS/Icon-76.png
BIN
media/iOS/Icon-76@2x.png
BIN
media/iOS/Icon-Small.png
BIN
media/iOS/Icon-Small@2x.png
BIN
media/iOS/Icon-Small@3x.png
BIN
media/iOS/iTunesArtwork.png
BIN
media/iOS/iTunesArtwork@2x.png
BIN
public/android-chrome-144x144.png
BIN
public/android-chrome-192x192.png
BIN
public/android-chrome-36x36.png
BIN
public/android-chrome-48x48.png
BIN
public/android-chrome-72x72.png
BIN
public/android-chrome-96x96.png
BIN
public/apple-touch-icon-114x114.png
BIN
public/apple-touch-icon-120x120.png
BIN
public/apple-touch-icon-144x144.png
BIN
public/apple-touch-icon-152x152.png
BIN
public/apple-touch-icon-180x180.png
BIN
public/apple-touch-icon-57x57.png
BIN
public/apple-touch-icon-60x60.png
BIN
public/apple-touch-icon-72x72.png
BIN
public/apple-touch-icon-76x76.png
+ 41
- 0
public/manifest.json
@@ -0,0 +1,41 @@ |
||
| 1 |
+{
|
|
| 2 |
+ "name": "Huginn", |
|
| 3 |
+ "icons": [ |
|
| 4 |
+ {
|
|
| 5 |
+ "src": "/android-chrome-36x36.png", |
|
| 6 |
+ "sizes": "36x36", |
|
| 7 |
+ "type": "image/png", |
|
| 8 |
+ "density": "0.75" |
|
| 9 |
+ }, |
|
| 10 |
+ {
|
|
| 11 |
+ "src": "/android-chrome-72x72.png", |
|
| 12 |
+ "sizes": "72x72", |
|
| 13 |
+ "type": "image/png", |
|
| 14 |
+ "density": "1.5" |
|
| 15 |
+ }, |
|
| 16 |
+ {
|
|
| 17 |
+ "src": "/android-chrome-144x144.png", |
|
| 18 |
+ "sizes": "144x144", |
|
| 19 |
+ "type": "image/png", |
|
| 20 |
+ "density": "3.0" |
|
| 21 |
+ }, |
|
| 22 |
+ {
|
|
| 23 |
+ "src": "/android-chrome-48x48.png", |
|
| 24 |
+ "sizes": "48x48", |
|
| 25 |
+ "type": "image/png", |
|
| 26 |
+ "density": "1.0" |
|
| 27 |
+ }, |
|
| 28 |
+ {
|
|
| 29 |
+ "src": "/android-chrome-96x96.png", |
|
| 30 |
+ "sizes": "96x96", |
|
| 31 |
+ "type": "image/png", |
|
| 32 |
+ "density": "2.0" |
|
| 33 |
+ }, |
|
| 34 |
+ {
|
|
| 35 |
+ "src": "/android-chrome-192x192.png", |
|
| 36 |
+ "sizes": "192x192", |
|
| 37 |
+ "type": "image/png", |
|
| 38 |
+ "density": "4.0" |
|
| 39 |
+ } |
|
| 40 |
+ ] |
|
| 41 |
+} |
+ 46
- 17
spec/concerns/dry_runnable_spec.rb
@@ -7,10 +7,22 @@ describe DryRunnable do |
||
| 7 | 7 |
can_dry_run! |
| 8 | 8 |
|
| 9 | 9 |
def check |
| 10 |
+ perform |
|
| 11 |
+ end |
|
| 12 |
+ |
|
| 13 |
+ def receive(events) |
|
| 14 |
+ events.each do |event| |
|
| 15 |
+ perform(event.payload['prefix']) |
|
| 16 |
+ end |
|
| 17 |
+ end |
|
| 18 |
+ |
|
| 19 |
+ private |
|
| 20 |
+ |
|
| 21 |
+ def perform(prefix = nil) |
|
| 10 | 22 |
log "Logging" |
| 11 |
- create_event payload: { 'test' => 'foo' }
|
|
| 23 |
+ create_event payload: { 'test' => "#{prefix}foo" }
|
|
| 12 | 24 |
error "Recording error" |
| 13 |
- create_event payload: { 'test' => 'bar' }
|
|
| 25 |
+ create_event payload: { 'test' => "#{prefix}bar" }
|
|
| 14 | 26 |
self.memory = { 'last_status' => 'ok', 'dry_run' => dry_run? }
|
| 15 | 27 |
save! |
| 16 | 28 |
end |
@@ -46,21 +58,6 @@ describe DryRunnable do |
||
| 46 | 58 |
expect(messages).to eq(['Logging', 'Recording error']) |
| 47 | 59 |
end |
| 48 | 60 |
|
| 49 |
- it "traps logging, event emission and memory updating, with dry_run? returning true" do |
|
| 50 |
- results = nil |
|
| 51 |
- |
|
| 52 |
- expect {
|
|
| 53 |
- results = @agent.dry_run! |
|
| 54 |
- @agent.reload |
|
| 55 |
- }.not_to change {
|
|
| 56 |
- [@agent.memory, counts] |
|
| 57 |
- } |
|
| 58 |
- |
|
| 59 |
- expect(results[:log]).to match(/\AI, .+ INFO -- : Logging\nE, .+ ERROR -- : Recording error\n/) |
|
| 60 |
- expect(results[:events]).to eq([{ 'test' => 'foo' }, { 'test' => 'bar' }])
|
|
| 61 |
- expect(results[:memory]).to eq({ 'last_status' => 'ok', 'dry_run' => true })
|
|
| 62 |
- end |
|
| 63 |
- |
|
| 64 | 61 |
it "does not perform dry-run if Agent does not support dry-run" do |
| 65 | 62 |
stub(@agent).can_dry_run? { false }
|
| 66 | 63 |
|
@@ -77,4 +74,36 @@ describe DryRunnable do |
||
| 77 | 74 |
expect(results[:events]).to eq([]) |
| 78 | 75 |
expect(results[:memory]).to eq({})
|
| 79 | 76 |
end |
| 77 |
+ |
|
| 78 |
+ describe "dry_run!" do |
|
| 79 |
+ it "traps any destructive operations during a run" do |
|
| 80 |
+ results = nil |
|
| 81 |
+ |
|
| 82 |
+ expect {
|
|
| 83 |
+ results = @agent.dry_run! |
|
| 84 |
+ @agent.reload |
|
| 85 |
+ }.not_to change {
|
|
| 86 |
+ [@agent.memory, counts] |
|
| 87 |
+ } |
|
| 88 |
+ |
|
| 89 |
+ expect(results[:log]).to match(/\AI, .+ INFO -- : Logging\nE, .+ ERROR -- : Recording error\n/) |
|
| 90 |
+ expect(results[:events]).to eq([{ 'test' => 'foo' }, { 'test' => 'bar' }])
|
|
| 91 |
+ expect(results[:memory]).to eq({ 'last_status' => 'ok', 'dry_run' => true })
|
|
| 92 |
+ end |
|
| 93 |
+ |
|
| 94 |
+ it "traps any destructive operations during a run when an event is given" do |
|
| 95 |
+ results = nil |
|
| 96 |
+ |
|
| 97 |
+ expect {
|
|
| 98 |
+ results = @agent.dry_run!(Event.new(payload: { 'prefix' => 'super' }))
|
|
| 99 |
+ @agent.reload |
|
| 100 |
+ }.not_to change {
|
|
| 101 |
+ [@agent.memory, counts] |
|
| 102 |
+ } |
|
| 103 |
+ |
|
| 104 |
+ expect(results[:log]).to match(/\AI, .+ INFO -- : Logging\nE, .+ ERROR -- : Recording error\n/) |
|
| 105 |
+ expect(results[:events]).to eq([{ 'test' => 'superfoo' }, { 'test' => 'superbar' }])
|
|
| 106 |
+ expect(results[:memory]).to eq({ 'last_status' => 'ok', 'dry_run' => true })
|
|
| 107 |
+ end |
|
| 108 |
+ end |
|
| 80 | 109 |
end |
+ 124
- 0
spec/concerns/liquid_interpolatable_spec.rb
@@ -38,6 +38,16 @@ describe LiquidInterpolatable::Filters do |
||
| 38 | 38 |
end |
| 39 | 39 |
end |
| 40 | 40 |
|
| 41 |
+ describe 'unescape' do |
|
| 42 |
+ let(:agent) { Agents::InterpolatableAgent.new(name: "test") }
|
|
| 43 |
+ |
|
| 44 |
+ it 'should unescape basic HTML entities' do |
|
| 45 |
+ agent.interpolation_context['something'] = ''<foo> & bar'' |
|
| 46 |
+ agent.options['cleaned'] = '{{ something | unescape }}'
|
|
| 47 |
+ expect(agent.interpolated['cleaned']).to eq("'<foo> & bar'")
|
|
| 48 |
+ end |
|
| 49 |
+ end |
|
| 50 |
+ |
|
| 41 | 51 |
describe 'to_xpath' do |
| 42 | 52 |
before do |
| 43 | 53 |
def @filter.to_xpath_roundtrip(string) |
@@ -96,4 +106,118 @@ describe LiquidInterpolatable::Filters do |
||
| 96 | 106 |
expect(@agent.interpolated['foo']).to eq('/dir/foo/index.html')
|
| 97 | 107 |
end |
| 98 | 108 |
end |
| 109 |
+ |
|
| 110 |
+ describe 'uri_expand' do |
|
| 111 |
+ before do |
|
| 112 |
+ stub_request(:head, 'https://t.co.x/aaaa'). |
|
| 113 |
+ to_return(status: 301, headers: { Location: 'https://bit.ly.x/bbbb' })
|
|
| 114 |
+ stub_request(:head, 'https://bit.ly.x/bbbb'). |
|
| 115 |
+ to_return(status: 301, headers: { Location: 'http://tinyurl.com.x/cccc' })
|
|
| 116 |
+ stub_request(:head, 'http://tinyurl.com.x/cccc'). |
|
| 117 |
+ to_return(status: 301, headers: { Location: 'http://www.example.com/welcome' })
|
|
| 118 |
+ stub_request(:head, 'http://www.example.com/welcome'). |
|
| 119 |
+ to_return(status: 200) |
|
| 120 |
+ |
|
| 121 |
+ (1..5).each do |i| |
|
| 122 |
+ stub_request(:head, "http://2many.x/#{i}").
|
|
| 123 |
+ to_return(status: 301, headers: { Location: "http://2many.x/#{i+1}" })
|
|
| 124 |
+ end |
|
| 125 |
+ stub_request(:head, 'http://2many.x/6'). |
|
| 126 |
+ to_return(status: 301, headers: { 'Content-Length' => '5' })
|
|
| 127 |
+ end |
|
| 128 |
+ |
|
| 129 |
+ it 'should handle inaccessible URIs' do |
|
| 130 |
+ expect(@filter.uri_expand(nil)).to eq('')
|
|
| 131 |
+ expect(@filter.uri_expand('')).to eq('')
|
|
| 132 |
+ expect(@filter.uri_expand(5)).to eq('5')
|
|
| 133 |
+ expect(@filter.uri_expand([])).to eq('[]')
|
|
| 134 |
+ expect(@filter.uri_expand({})).to eq('{}')
|
|
| 135 |
+ expect(@filter.uri_expand(URI('/'))).to eq('/')
|
|
| 136 |
+ expect(@filter.uri_expand(URI('http:google.com'))).to eq('http:google.com')
|
|
| 137 |
+ expect(@filter.uri_expand(URI('http:/google.com'))).to eq('http:/google.com')
|
|
| 138 |
+ expect(@filter.uri_expand(URI('ftp://ftp.freebsd.org/pub/FreeBSD/README.TXT'))).to eq('ftp://ftp.freebsd.org/pub/FreeBSD/README.TXT')
|
|
| 139 |
+ end |
|
| 140 |
+ |
|
| 141 |
+ it 'should follow redirects' do |
|
| 142 |
+ expect(@filter.uri_expand('https://t.co.x/aaaa')).to eq('http://www.example.com/welcome')
|
|
| 143 |
+ end |
|
| 144 |
+ |
|
| 145 |
+ it 'should respect the limit for the number of redirects' do |
|
| 146 |
+ expect(@filter.uri_expand('http://2many.x/1')).to eq('http://2many.x/1')
|
|
| 147 |
+ expect(@filter.uri_expand('http://2many.x/1', 6)).to eq('http://2many.x/6')
|
|
| 148 |
+ end |
|
| 149 |
+ |
|
| 150 |
+ it 'should detect a redirect loop' do |
|
| 151 |
+ stub_request(:head, 'http://bad.x/aaaa'). |
|
| 152 |
+ to_return(status: 301, headers: { Location: 'http://bad.x/bbbb' })
|
|
| 153 |
+ stub_request(:head, 'http://bad.x/bbbb'). |
|
| 154 |
+ to_return(status: 301, headers: { Location: 'http://bad.x/aaaa' })
|
|
| 155 |
+ |
|
| 156 |
+ expect(@filter.uri_expand('http://bad.x/aaaa')).to eq('http://bad.x/aaaa')
|
|
| 157 |
+ end |
|
| 158 |
+ |
|
| 159 |
+ it 'should be able to handle an FTP URL' do |
|
| 160 |
+ stub_request(:head, 'http://downloads.x/aaaa'). |
|
| 161 |
+ to_return(status: 301, headers: { Location: 'http://downloads.x/download?file=aaaa.zip' })
|
|
| 162 |
+ stub_request(:head, 'http://downloads.x/download'). |
|
| 163 |
+ with(query: { file: 'aaaa.zip' }).
|
|
| 164 |
+ to_return(status: 301, headers: { Location: 'ftp://downloads.x/pub/aaaa.zip' })
|
|
| 165 |
+ |
|
| 166 |
+ expect(@filter.uri_expand('http://downloads.x/aaaa')).to eq('ftp://downloads.x/pub/aaaa.zip')
|
|
| 167 |
+ end |
|
| 168 |
+ |
|
| 169 |
+ describe 'used in interpolation' do |
|
| 170 |
+ before do |
|
| 171 |
+ @agent = Agents::InterpolatableAgent.new(name: "test") |
|
| 172 |
+ end |
|
| 173 |
+ |
|
| 174 |
+ it 'should follow redirects' do |
|
| 175 |
+ @agent.interpolation_context['short_url'] = 'https://t.co.x/aaaa' |
|
| 176 |
+ @agent.options['long_url'] = '{{ short_url | uri_expand }}'
|
|
| 177 |
+ expect(@agent.interpolated['long_url']).to eq('http://www.example.com/welcome')
|
|
| 178 |
+ end |
|
| 179 |
+ |
|
| 180 |
+ it 'should respect the limit for the number of redirects' do |
|
| 181 |
+ @agent.interpolation_context['short_url'] = 'http://2many.x/1' |
|
| 182 |
+ @agent.options['long_url'] = '{{ short_url | uri_expand }}'
|
|
| 183 |
+ expect(@agent.interpolated['long_url']).to eq('http://2many.x/1')
|
|
| 184 |
+ |
|
| 185 |
+ @agent.interpolation_context['short_url'] = 'http://2many.x/1' |
|
| 186 |
+ @agent.options['long_url'] = '{{ short_url | uri_expand:6 }}'
|
|
| 187 |
+ expect(@agent.interpolated['long_url']).to eq('http://2many.x/6')
|
|
| 188 |
+ end |
|
| 189 |
+ end |
|
| 190 |
+ end |
|
| 191 |
+ |
|
| 192 |
+ describe 'regex_replace_first' do |
|
| 193 |
+ let(:agent) { Agents::InterpolatableAgent.new(name: "test") }
|
|
| 194 |
+ |
|
| 195 |
+ it 'should replace the first occurrence of a string using regex' do |
|
| 196 |
+ agent.interpolation_context['something'] = 'foobar foobar' |
|
| 197 |
+ agent.options['cleaned'] = '{{ something | regex_replace_first: "\S+bar", "foobaz" }}'
|
|
| 198 |
+ expect(agent.interpolated['cleaned']).to eq('foobaz foobar')
|
|
| 199 |
+ end |
|
| 200 |
+ |
|
| 201 |
+ it 'should support escaped characters' do |
|
| 202 |
+ agent.interpolation_context['something'] = "foo\\1\n\nfoo\\bar\n\nfoo\\baz" |
|
| 203 |
+ agent.options['test'] = "{{ something | regex_replace_first: '\\\\(\\w{2,})', '\\1\\\\' | regex_replace_first: '\\n+', '\\n' }}"
|
|
| 204 |
+ expect(agent.interpolated['test']).to eq("foo\\1\nfoobar\\\n\nfoo\\baz")
|
|
| 205 |
+ end |
|
| 206 |
+ end |
|
| 207 |
+ |
|
| 208 |
+ describe 'regex_replace' do |
|
| 209 |
+ let(:agent) { Agents::InterpolatableAgent.new(name: "test") }
|
|
| 210 |
+ |
|
| 211 |
+ it 'should replace the all occurrences of a string using regex' do |
|
| 212 |
+ agent.interpolation_context['something'] = 'foobar foobar' |
|
| 213 |
+ agent.options['cleaned'] = '{{ something | regex_replace: "\S+bar", "foobaz" }}'
|
|
| 214 |
+ expect(agent.interpolated['cleaned']).to eq('foobaz foobaz')
|
|
| 215 |
+ end |
|
| 216 |
+ |
|
| 217 |
+ it 'should support escaped characters' do |
|
| 218 |
+ agent.interpolation_context['something'] = "foo\\1\n\nfoo\\bar\n\nfoo\\baz" |
|
| 219 |
+ agent.options['test'] = "{{ something | regex_replace: '\\\\(\\w{2,})', '\\1\\\\' | regex_replace: '\\n+', '\\n' }}"
|
|
| 220 |
+ expect(agent.interpolated['test']).to eq("foo\\1\nfoobar\\\nfoobaz\\")
|
|
| 221 |
+ end |
|
| 222 |
+ end |
|
| 99 | 223 |
end |
+ 264
- 0
spec/concerns/sortable_events_spec.rb
@@ -0,0 +1,264 @@ |
||
| 1 |
+require 'spec_helper' |
|
| 2 |
+ |
|
| 3 |
+describe SortableEvents do |
|
| 4 |
+ let(:agent_class) {
|
|
| 5 |
+ Class.new(Agent) do |
|
| 6 |
+ include SortableEvents |
|
| 7 |
+ |
|
| 8 |
+ default_schedule 'never' |
|
| 9 |
+ |
|
| 10 |
+ def self.valid_type?(name) |
|
| 11 |
+ true |
|
| 12 |
+ end |
|
| 13 |
+ end |
|
| 14 |
+ } |
|
| 15 |
+ |
|
| 16 |
+ def new_agent(events_order = nil) |
|
| 17 |
+ options = {}
|
|
| 18 |
+ options['events_order'] = events_order if events_order |
|
| 19 |
+ agent_class.new(name: 'test', options: options) { |agent|
|
|
| 20 |
+ agent.user = users(:bob) |
|
| 21 |
+ } |
|
| 22 |
+ end |
|
| 23 |
+ |
|
| 24 |
+ describe 'validations' do |
|
| 25 |
+ let(:agent_class) {
|
|
| 26 |
+ Class.new(Agent) do |
|
| 27 |
+ include SortableEvents |
|
| 28 |
+ |
|
| 29 |
+ default_schedule 'never' |
|
| 30 |
+ |
|
| 31 |
+ def self.valid_type?(name) |
|
| 32 |
+ true |
|
| 33 |
+ end |
|
| 34 |
+ end |
|
| 35 |
+ } |
|
| 36 |
+ |
|
| 37 |
+ def new_agent(events_order = nil) |
|
| 38 |
+ options = {}
|
|
| 39 |
+ options['events_order'] = events_order if events_order |
|
| 40 |
+ agent_class.new(name: 'test', options: options) { |agent|
|
|
| 41 |
+ agent.user = users(:bob) |
|
| 42 |
+ } |
|
| 43 |
+ end |
|
| 44 |
+ |
|
| 45 |
+ it 'should allow events_order to be unspecified, null or an empty array' do |
|
| 46 |
+ expect(new_agent()).to be_valid |
|
| 47 |
+ expect(new_agent(nil)).to be_valid |
|
| 48 |
+ expect(new_agent([])).to be_valid |
|
| 49 |
+ end |
|
| 50 |
+ |
|
| 51 |
+ it 'should not allow events_order to be a non-array object' do |
|
| 52 |
+ agent = new_agent(0) |
|
| 53 |
+ expect(agent).not_to be_valid |
|
| 54 |
+ expect(agent.errors[:base]).to include(/events_order/) |
|
| 55 |
+ |
|
| 56 |
+ agent = new_agent('')
|
|
| 57 |
+ expect(agent).not_to be_valid |
|
| 58 |
+ expect(agent.errors[:base]).to include(/events_order/) |
|
| 59 |
+ |
|
| 60 |
+ agent = new_agent({})
|
|
| 61 |
+ expect(agent).not_to be_valid |
|
| 62 |
+ expect(agent.errors[:base]).to include(/events_order/) |
|
| 63 |
+ end |
|
| 64 |
+ |
|
| 65 |
+ it 'should not allow events_order to be an array containing unexpected objects' do |
|
| 66 |
+ agent = new_agent(['{{key}}', 1])
|
|
| 67 |
+ expect(agent).not_to be_valid |
|
| 68 |
+ expect(agent.errors[:base]).to include(/events_order/) |
|
| 69 |
+ |
|
| 70 |
+ agent = new_agent(['{{key1}}', ['{{key2}}', 'unknown']])
|
|
| 71 |
+ expect(agent).not_to be_valid |
|
| 72 |
+ expect(agent.errors[:base]).to include(/events_order/) |
|
| 73 |
+ end |
|
| 74 |
+ |
|
| 75 |
+ it 'should allow events_order to be an array containing strings and valid tuples' do |
|
| 76 |
+ agent = new_agent(['{{key1}}', ['{{key2}}'], ['{{key3}}', 'number']])
|
|
| 77 |
+ expect(agent).to be_valid |
|
| 78 |
+ |
|
| 79 |
+ agent = new_agent(['{{key1}}', ['{{key2}}'], ['{{key3}}', 'number'], ['{{key4}}', 'time', true]])
|
|
| 80 |
+ expect(agent).to be_valid |
|
| 81 |
+ end |
|
| 82 |
+ end |
|
| 83 |
+ |
|
| 84 |
+ describe 'sort_events' do |
|
| 85 |
+ let(:payloads) {
|
|
| 86 |
+ [ |
|
| 87 |
+ { 'title' => 'TitleA', 'score' => 4, 'updated_on' => '7 Jul 2015' },
|
|
| 88 |
+ { 'title' => 'TitleB', 'score' => 2, 'updated_on' => '25 Jun 2014' },
|
|
| 89 |
+ { 'title' => 'TitleD', 'score' => 10, 'updated_on' => '10 Jan 2015' },
|
|
| 90 |
+ { 'title' => 'TitleC', 'score' => 10, 'updated_on' => '9 Feb 2015' },
|
|
| 91 |
+ ] |
|
| 92 |
+ } |
|
| 93 |
+ |
|
| 94 |
+ let(:events) {
|
|
| 95 |
+ payloads.map { |payload| Event.new(payload: payload) }
|
|
| 96 |
+ } |
|
| 97 |
+ |
|
| 98 |
+ it 'should sort events by a given key' do |
|
| 99 |
+ agent = new_agent(['{{title}}'])
|
|
| 100 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleC TitleD])
|
|
| 101 |
+ |
|
| 102 |
+ agent = new_agent([['{{title}}', 'string', true]])
|
|
| 103 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleD TitleC TitleB TitleA])
|
|
| 104 |
+ end |
|
| 105 |
+ |
|
| 106 |
+ it 'should sort events by multiple keys' do |
|
| 107 |
+ agent = new_agent([['{{score}}', 'number'], '{{title}}'])
|
|
| 108 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleA TitleC TitleD])
|
|
| 109 |
+ |
|
| 110 |
+ agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
|
|
| 111 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleA TitleD TitleC])
|
|
| 112 |
+ end |
|
| 113 |
+ |
|
| 114 |
+ it 'should sort events by time' do |
|
| 115 |
+ agent = new_agent([['{{updated_on}}', 'time']])
|
|
| 116 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleB TitleD TitleC TitleA])
|
|
| 117 |
+ end |
|
| 118 |
+ |
|
| 119 |
+ it 'should sort events stably' do |
|
| 120 |
+ agent = new_agent(['<constant>']) |
|
| 121 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
|
|
| 122 |
+ |
|
| 123 |
+ agent = new_agent([['<constant>', 'string', true]]) |
|
| 124 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
|
|
| 125 |
+ end |
|
| 126 |
+ |
|
| 127 |
+ it 'should support _index_' do |
|
| 128 |
+ agent = new_agent([['{{_index_}}', 'number', true]])
|
|
| 129 |
+ expect(agent.__send__(:sort_events, events).map { |e| e.payload['title'] }).to eq(%w[TitleC TitleD TitleB TitleA])
|
|
| 130 |
+ end |
|
| 131 |
+ end |
|
| 132 |
+ |
|
| 133 |
+ describe 'automatic event sorter' do |
|
| 134 |
+ describe 'declaration' do |
|
| 135 |
+ let(:passive_agent_class) {
|
|
| 136 |
+ Class.new(Agent) do |
|
| 137 |
+ include SortableEvents |
|
| 138 |
+ |
|
| 139 |
+ cannot_create_events! |
|
| 140 |
+ end |
|
| 141 |
+ } |
|
| 142 |
+ |
|
| 143 |
+ let(:active_agent_class) {
|
|
| 144 |
+ Class.new(Agent) do |
|
| 145 |
+ include SortableEvents |
|
| 146 |
+ end |
|
| 147 |
+ } |
|
| 148 |
+ |
|
| 149 |
+ describe 'can_order_created_events!' do |
|
| 150 |
+ it 'should refuse to work if called from an Agent that cannot create events' do |
|
| 151 |
+ expect {
|
|
| 152 |
+ passive_agent_class.class_eval do |
|
| 153 |
+ can_order_created_events! |
|
| 154 |
+ end |
|
| 155 |
+ }.to raise_error |
|
| 156 |
+ end |
|
| 157 |
+ |
|
| 158 |
+ it 'should work if called from an Agent that can create events' do |
|
| 159 |
+ expect {
|
|
| 160 |
+ active_agent_class.class_eval do |
|
| 161 |
+ can_order_created_events! |
|
| 162 |
+ end |
|
| 163 |
+ }.not_to raise_error |
|
| 164 |
+ end |
|
| 165 |
+ end |
|
| 166 |
+ |
|
| 167 |
+ describe 'can_order_created_events?' do |
|
| 168 |
+ it 'should return false unless an Agent declares can_order_created_events!' do |
|
| 169 |
+ expect(active_agent_class.can_order_created_events?).to eq(false) |
|
| 170 |
+ expect(active_agent_class.new.can_order_created_events?).to eq(false) |
|
| 171 |
+ end |
|
| 172 |
+ |
|
| 173 |
+ it 'should return true if an Agent declares can_order_created_events!' do |
|
| 174 |
+ active_agent_class.class_eval do |
|
| 175 |
+ can_order_created_events! |
|
| 176 |
+ end |
|
| 177 |
+ |
|
| 178 |
+ expect(active_agent_class.can_order_created_events?).to eq(true) |
|
| 179 |
+ expect(active_agent_class.new.can_order_created_events?).to eq(true) |
|
| 180 |
+ end |
|
| 181 |
+ end |
|
| 182 |
+ end |
|
| 183 |
+ |
|
| 184 |
+ describe 'behavior' do |
|
| 185 |
+ class Agents::EventOrderableAgent < Agent |
|
| 186 |
+ include SortableEvents |
|
| 187 |
+ |
|
| 188 |
+ default_schedule 'never' |
|
| 189 |
+ |
|
| 190 |
+ can_order_created_events! |
|
| 191 |
+ |
|
| 192 |
+ attr_accessor :payloads_to_emit |
|
| 193 |
+ |
|
| 194 |
+ def self.valid_type?(name) |
|
| 195 |
+ true |
|
| 196 |
+ end |
|
| 197 |
+ |
|
| 198 |
+ def check |
|
| 199 |
+ payloads_to_emit.each do |payload| |
|
| 200 |
+ create_event payload: payload |
|
| 201 |
+ end |
|
| 202 |
+ end |
|
| 203 |
+ |
|
| 204 |
+ def receive(events) |
|
| 205 |
+ events.each do |event| |
|
| 206 |
+ payloads_to_emit.each do |payload| |
|
| 207 |
+ create_event payload: payload.merge('title' => payload['title'] + event.payload['title_suffix'])
|
|
| 208 |
+ end |
|
| 209 |
+ end |
|
| 210 |
+ end |
|
| 211 |
+ end |
|
| 212 |
+ |
|
| 213 |
+ def new_agent(events_order = nil) |
|
| 214 |
+ options = {}
|
|
| 215 |
+ options['events_order'] = events_order if events_order |
|
| 216 |
+ Agents::EventOrderableAgent.new(name: 'test', options: options) { |agent|
|
|
| 217 |
+ agent.user = users(:bob) |
|
| 218 |
+ agent.payloads_to_emit = payloads |
|
| 219 |
+ } |
|
| 220 |
+ end |
|
| 221 |
+ |
|
| 222 |
+ let(:payloads) {
|
|
| 223 |
+ [ |
|
| 224 |
+ { 'title' => 'TitleA', 'score' => 4, 'updated_on' => '7 Jul 2015' },
|
|
| 225 |
+ { 'title' => 'TitleB', 'score' => 2, 'updated_on' => '25 Jun 2014' },
|
|
| 226 |
+ { 'title' => 'TitleD', 'score' => 10, 'updated_on' => '10 Jan 2015' },
|
|
| 227 |
+ { 'title' => 'TitleC', 'score' => 10, 'updated_on' => '9 Feb 2015' },
|
|
| 228 |
+ ] |
|
| 229 |
+ } |
|
| 230 |
+ |
|
| 231 |
+ it 'should keep the order of created events unless events_order is specified' do |
|
| 232 |
+ [[], [nil], [[]]].each do |args| |
|
| 233 |
+ agent = new_agent(*args) |
|
| 234 |
+ agent.save! |
|
| 235 |
+ expect { agent.check }.to change { Event.count }.by(4)
|
|
| 236 |
+ events = agent.events.last(4).sort_by(&:id) |
|
| 237 |
+ expect(events.map { |event| event.payload['title'] }).to eq(%w[TitleA TitleB TitleD TitleC])
|
|
| 238 |
+ end |
|
| 239 |
+ end |
|
| 240 |
+ |
|
| 241 |
+ it 'should sort events created in check() in the order specified in events_order' do |
|
| 242 |
+ agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
|
|
| 243 |
+ agent.save! |
|
| 244 |
+ expect { agent.check }.to change { Event.count }.by(4)
|
|
| 245 |
+ events = agent.events.last(4).sort_by(&:id) |
|
| 246 |
+ expect(events.map { |event| event.payload['title'] }).to eq(%w[TitleB TitleA TitleD TitleC])
|
|
| 247 |
+ end |
|
| 248 |
+ |
|
| 249 |
+ it 'should sort events created in receive() in the order specified in events_order' do |
|
| 250 |
+ agent = new_agent([['{{score}}', 'number'], ['{{title}}', 'string', true]])
|
|
| 251 |
+ agent.save! |
|
| 252 |
+ expect {
|
|
| 253 |
+ agent.receive([Event.new(payload: { 'title_suffix' => ' [new]' }),
|
|
| 254 |
+ Event.new(payload: { 'title_suffix' => ' [popular]' })])
|
|
| 255 |
+ }.to change { Event.count }.by(8)
|
|
| 256 |
+ events = agent.events.last(8).sort_by(&:id) |
|
| 257 |
+ expect(events.map { |event| event.payload['title'] }).to eq([
|
|
| 258 |
+ 'TitleB [new]', 'TitleA [new]', 'TitleD [new]', 'TitleC [new]', |
|
| 259 |
+ 'TitleB [popular]', 'TitleA [popular]', 'TitleD [popular]', 'TitleC [popular]', |
|
| 260 |
+ ]) |
|
| 261 |
+ end |
|
| 262 |
+ end |
|
| 263 |
+ end |
|
| 264 |
+end |
+ 44
- 11
spec/controllers/agents_controller_spec.rb
@@ -87,19 +87,35 @@ describe AgentsController do |
||
| 87 | 87 |
end |
| 88 | 88 |
end |
| 89 | 89 |
|
| 90 |
- describe "GET new with :id" do |
|
| 91 |
- it "opens a clone of a given Agent" do |
|
| 92 |
- sign_in users(:bob) |
|
| 93 |
- get :new, :id => agents(:bob_website_agent).to_param |
|
| 94 |
- expect(assigns(:agent).attributes).to eq(users(:bob).agents.build_clone(agents(:bob_website_agent)).attributes) |
|
| 90 |
+ describe "GET new" do |
|
| 91 |
+ describe "with :id" do |
|
| 92 |
+ it "opens a clone of a given Agent" do |
|
| 93 |
+ sign_in users(:bob) |
|
| 94 |
+ get :new, :id => agents(:bob_website_agent).to_param |
|
| 95 |
+ expect(assigns(:agent).attributes).to eq(users(:bob).agents.build_clone(agents(:bob_website_agent)).attributes) |
|
| 96 |
+ end |
|
| 97 |
+ |
|
| 98 |
+ it "only allows the current user to clone his own Agent" do |
|
| 99 |
+ sign_in users(:bob) |
|
| 100 |
+ |
|
| 101 |
+ expect {
|
|
| 102 |
+ get :new, :id => agents(:jane_website_agent).to_param |
|
| 103 |
+ }.to raise_error(ActiveRecord::RecordNotFound) |
|
| 104 |
+ end |
|
| 95 | 105 |
end |
| 96 | 106 |
|
| 97 |
- it "only allows the current user to clone his own Agent" do |
|
| 98 |
- sign_in users(:bob) |
|
| 107 |
+ describe "with a scenario_id" do |
|
| 108 |
+ it 'populates the assigned agent with the scenario' do |
|
| 109 |
+ sign_in users(:bob) |
|
| 110 |
+ get :new, :scenario_id => scenarios(:bob_weather).id |
|
| 111 |
+ expect(assigns(:agent).scenario_ids).to eq([scenarios(:bob_weather).id]) |
|
| 112 |
+ end |
|
| 99 | 113 |
|
| 100 |
- expect {
|
|
| 101 |
- get :new, :id => agents(:jane_website_agent).to_param |
|
| 102 |
- }.to raise_error(ActiveRecord::RecordNotFound) |
|
| 114 |
+ it "does not see other user's scenarios" do |
|
| 115 |
+ sign_in users(:bob) |
|
| 116 |
+ get :new, :scenario_id => scenarios(:jane_weather).id |
|
| 117 |
+ expect(assigns(:agent).scenario_ids).to eq([]) |
|
| 118 |
+ end |
|
| 103 | 119 |
end |
| 104 | 120 |
end |
| 105 | 121 |
|
@@ -349,6 +365,10 @@ describe AgentsController do |
||
| 349 | 365 |
end |
| 350 | 366 |
|
| 351 | 367 |
describe "POST dry_run" do |
| 368 |
+ before do |
|
| 369 |
+ stub_request(:any, /xkcd/).to_return(body: File.read(Rails.root.join("spec/data_fixtures/xkcd.html")), status: 200)
|
|
| 370 |
+ end |
|
| 371 |
+ |
|
| 352 | 372 |
it "does not actually create any agent, event or log" do |
| 353 | 373 |
sign_in users(:bob) |
| 354 | 374 |
expect {
|
@@ -368,11 +388,24 @@ describe AgentsController do |
||
| 368 | 388 |
sign_in users(:bob) |
| 369 | 389 |
agent = agents(:bob_weather_agent) |
| 370 | 390 |
expect {
|
| 371 |
- post :dry_run, id: agents(:bob_website_agent), agent: valid_attributes(name: 'New Name') |
|
| 391 |
+ post :dry_run, id: agent, agent: valid_attributes(name: 'New Name') |
|
| 372 | 392 |
}.not_to change {
|
| 373 | 393 |
[users(:bob).agents.count, users(:bob).events.count, users(:bob).logs.count, agent.name, agent.updated_at] |
| 374 | 394 |
} |
| 375 | 395 |
end |
| 396 |
+ |
|
| 397 |
+ it "accepts an event" do |
|
| 398 |
+ sign_in users(:bob) |
|
| 399 |
+ agent = agents(:bob_website_agent) |
|
| 400 |
+ url_from_event = "http://xkcd.com/?from_event=1".freeze |
|
| 401 |
+ expect {
|
|
| 402 |
+ post :dry_run, id: agent, event: { url: url_from_event }
|
|
| 403 |
+ }.not_to change {
|
|
| 404 |
+ [users(:bob).agents.count, users(:bob).events.count, users(:bob).logs.count, agent.name, agent.updated_at] |
|
| 405 |
+ } |
|
| 406 |
+ json = JSON.parse(response.body) |
|
| 407 |
+ expect(json['log']).to match(/^I, .* : Fetching #{Regexp.quote(url_from_event)}$/)
|
|
| 408 |
+ end |
|
| 376 | 409 |
end |
| 377 | 410 |
|
| 378 | 411 |
describe "DELETE memory" do |
+ 16
- 3
spec/controllers/jobs_controller_spec.rb
@@ -1,7 +1,6 @@ |
||
| 1 | 1 |
require 'spec_helper' |
| 2 | 2 |
|
| 3 | 3 |
describe JobsController do |
| 4 |
- |
|
| 5 | 4 |
describe "GET index" do |
| 6 | 5 |
before do |
| 7 | 6 |
async_handler_yaml = |
@@ -71,12 +70,26 @@ describe JobsController do |
||
| 71 | 70 |
before do |
| 72 | 71 |
@failed = Delayed::Job.create(failed_at: Time.now - 1.minute) |
| 73 | 72 |
@running = Delayed::Job.create(locked_at: Time.now, locked_by: 'test') |
| 73 |
+ @pending = Delayed::Job.create |
|
| 74 | 74 |
sign_in users(:jane) |
| 75 | 75 |
end |
| 76 | 76 |
|
| 77 | 77 |
it "just destroy failed jobs" do |
| 78 |
- expect { delete :destroy_failed, id: @failed.id }.to change(Delayed::Job, :count).by(-1)
|
|
| 79 |
- expect { delete :destroy_failed, id: @running.id }.to change(Delayed::Job, :count).by(0)
|
|
| 78 |
+ expect { delete :destroy_failed }.to change(Delayed::Job, :count).by(-1)
|
|
| 79 |
+ end |
|
| 80 |
+ end |
|
| 81 |
+ |
|
| 82 |
+ describe "DELETE destroy_all" do |
|
| 83 |
+ before do |
|
| 84 |
+ @failed = Delayed::Job.create(failed_at: Time.now - 1.minute) |
|
| 85 |
+ @running = Delayed::Job.create(locked_at: Time.now, locked_by: 'test') |
|
| 86 |
+ @pending = Delayed::Job.create |
|
| 87 |
+ sign_in users(:jane) |
|
| 88 |
+ end |
|
| 89 |
+ |
|
| 90 |
+ it "destroys all jobs" do |
|
| 91 |
+ expect { delete :destroy_all }.to change(Delayed::Job, :count).by(-2)
|
|
| 92 |
+ expect(Delayed::Job.find(@running.id)).to be |
|
| 80 | 93 |
end |
| 81 | 94 |
end |
| 82 | 95 |
end |
+ 2
- 2
spec/fixtures/agents.yml
@@ -38,7 +38,7 @@ bob_weather_agent: |
||
| 38 | 38 |
schedule: "midnight" |
| 39 | 39 |
name: "SF Weather" |
| 40 | 40 |
guid: <%= SecureRandom.hex %> |
| 41 |
- keep_events_for: 45 |
|
| 41 |
+ keep_events_for: <%= 45.days %> |
|
| 42 | 42 |
options: <%= { :location => 94102, :lat => 37.779329, :lng => -122.41915, :api_key => 'test' }.to_json.inspect %>
|
| 43 | 43 |
|
| 44 | 44 |
jane_weather_agent: |
@@ -47,7 +47,7 @@ jane_weather_agent: |
||
| 47 | 47 |
schedule: "midnight" |
| 48 | 48 |
name: "SF Weather" |
| 49 | 49 |
guid: <%= SecureRandom.hex %> |
| 50 |
- keep_events_for: 30 |
|
| 50 |
+ keep_events_for: <%= 30.days %> |
|
| 51 | 51 |
options: <%= { :location => 94103, :lat => 37.779329, :lng => -122.41915, :api_key => 'test' }.to_json.inspect %>
|
| 52 | 52 |
|
| 53 | 53 |
jane_rain_notifier_agent: |
+ 9
- 0
spec/lib/agents_exporter_spec.rb
@@ -21,10 +21,12 @@ describe AgentsExporter do |
||
| 21 | 21 |
expect(data[:description]).to eq(description) |
| 22 | 22 |
expect(data[:source_url]).to eq(source_url) |
| 23 | 23 |
expect(data[:guid]).to eq(guid) |
| 24 |
+ expect(data[:schema_version]).to eq(1) |
|
| 24 | 25 |
expect(data[:tag_fg_color]).to eq(tag_fg_color) |
| 25 | 26 |
expect(data[:tag_bg_color]).to eq(tag_bg_color) |
| 26 | 27 |
expect(Time.parse(data[:exported_at])).to be_within(2).of(Time.now.utc) |
| 27 | 28 |
expect(data[:links]).to eq([{ :source => 0, :receiver => 1 }])
|
| 29 |
+ expect(data[:control_links]).to eq([]) |
|
| 28 | 30 |
expect(data[:agents]).to eq(agent_list.map { |agent| exporter.agent_as_json(agent) })
|
| 29 | 31 |
expect(data[:agents].all? { |agent_json| agent_json[:guid].present? && agent_json[:type].present? && agent_json[:name].present? }).to be_truthy
|
| 30 | 32 |
|
@@ -38,6 +40,13 @@ describe AgentsExporter do |
||
| 38 | 40 |
|
| 39 | 41 |
expect(exporter.as_json[:links]).to eq([{ :source => 0, :receiver => 1 }])
|
| 40 | 42 |
end |
| 43 |
+ |
|
| 44 |
+ it "outputs control links to agents within the incoming set, but not outside it" do |
|
| 45 |
+ agents(:jane_rain_notifier_agent).control_targets = [agents(:jane_weather_agent), agents(:jane_basecamp_agent)] |
|
| 46 |
+ agents(:jane_rain_notifier_agent).save! |
|
| 47 |
+ |
|
| 48 |
+ expect(exporter.as_json[:control_links]).to eq([{ :controller => 1, :control_target => 0 }])
|
|
| 49 |
+ end |
|
| 41 | 50 |
end |
| 42 | 51 |
|
| 43 | 52 |
describe "#filename" do |
+ 58
- 0
spec/lib/utils_spec.rb
@@ -114,4 +114,62 @@ describe Utils do |
||
| 114 | 114 |
expect(cleaned_json).to include("<\\/script>")
|
| 115 | 115 |
end |
| 116 | 116 |
end |
| 117 |
+ |
|
| 118 |
+ describe "#sort_tuples!" do |
|
| 119 |
+ let(:tuples) {
|
|
| 120 |
+ time = Time.now |
|
| 121 |
+ [ |
|
| 122 |
+ [2, "a", time - 1], # 0 |
|
| 123 |
+ [2, "b", time - 1], # 1 |
|
| 124 |
+ [1, "b", time - 1], # 2 |
|
| 125 |
+ [1, "b", time], # 3 |
|
| 126 |
+ [1, "a", time], # 4 |
|
| 127 |
+ [2, "a", time + 1], # 5 |
|
| 128 |
+ [2, "a", time], # 6 |
|
| 129 |
+ ] |
|
| 130 |
+ } |
|
| 131 |
+ |
|
| 132 |
+ it "sorts tuples like arrays by default" do |
|
| 133 |
+ expected = tuples.values_at(4, 2, 3, 0, 6, 5, 1) |
|
| 134 |
+ |
|
| 135 |
+ Utils.sort_tuples!(tuples) |
|
| 136 |
+ expect(tuples).to eq expected |
|
| 137 |
+ end |
|
| 138 |
+ |
|
| 139 |
+ it "sorts tuples in order specified: case 1" do |
|
| 140 |
+ # order by x1 asc, x2 desc, c3 asc |
|
| 141 |
+ orders = [false, true, false] |
|
| 142 |
+ expected = tuples.values_at(2, 3, 4, 1, 0, 6, 5) |
|
| 143 |
+ |
|
| 144 |
+ Utils.sort_tuples!(tuples, orders) |
|
| 145 |
+ expect(tuples).to eq expected |
|
| 146 |
+ end |
|
| 147 |
+ |
|
| 148 |
+ it "sorts tuples in order specified: case 2" do |
|
| 149 |
+ # order by x1 desc, x2 asc, c3 desc |
|
| 150 |
+ orders = [true, false, true] |
|
| 151 |
+ expected = tuples.values_at(5, 6, 0, 1, 4, 3, 2) |
|
| 152 |
+ |
|
| 153 |
+ Utils.sort_tuples!(tuples, orders) |
|
| 154 |
+ expect(tuples).to eq expected |
|
| 155 |
+ end |
|
| 156 |
+ |
|
| 157 |
+ it "always succeeds in sorting even if it finds pairs of incomparable objects" do |
|
| 158 |
+ time = Time.now |
|
| 159 |
+ tuples = [ |
|
| 160 |
+ [2, "a", time - 1], # 0 |
|
| 161 |
+ [1, "b", nil], # 1 |
|
| 162 |
+ [1, "b", time], # 2 |
|
| 163 |
+ ["2", nil, time], # 3 |
|
| 164 |
+ [1, nil, time], # 4 |
|
| 165 |
+ [nil, "a", time + 1], # 5 |
|
| 166 |
+ [2, "a", time], # 6 |
|
| 167 |
+ ] |
|
| 168 |
+ orders = [true, false, true] |
|
| 169 |
+ expected = tuples.values_at(3, 6, 0, 4, 2, 1, 5) |
|
| 170 |
+ |
|
| 171 |
+ Utils.sort_tuples!(tuples, orders) |
|
| 172 |
+ expect(tuples).to eq expected |
|
| 173 |
+ end |
|
| 174 |
+ end |
|
| 117 | 175 |
end |
+ 11
- 11
spec/models/agent_spec.rb
@@ -546,11 +546,11 @@ describe Agent do |
||
| 546 | 546 |
expect(agent).to have(1).errors_on(:keep_events_for) |
| 547 | 547 |
agent.keep_events_for = "" |
| 548 | 548 |
expect(agent).to have(1).errors_on(:keep_events_for) |
| 549 |
- agent.keep_events_for = 5 |
|
| 549 |
+ agent.keep_events_for = 5.days.to_i |
|
| 550 | 550 |
expect(agent).to be_valid |
| 551 | 551 |
agent.keep_events_for = 0 |
| 552 | 552 |
expect(agent).to be_valid |
| 553 |
- agent.keep_events_for = 365 |
|
| 553 |
+ agent.keep_events_for = 365.days.to_i |
|
| 554 | 554 |
expect(agent).to be_valid |
| 555 | 555 |
|
| 556 | 556 |
# Rails seems to call to_i on the input. This guards against future changes to that behavior. |
@@ -564,7 +564,7 @@ describe Agent do |
||
| 564 | 564 |
@time = "2014-01-01 01:00:00 +00:00" |
| 565 | 565 |
time_travel_to @time do |
| 566 | 566 |
@agent = Agents::SomethingSource.new(:name => "something") |
| 567 |
- @agent.keep_events_for = 5 |
|
| 567 |
+ @agent.keep_events_for = 5.days |
|
| 568 | 568 |
@agent.user = users(:bob) |
| 569 | 569 |
@agent.save! |
| 570 | 570 |
@event = @agent.create_event :payload => { "hello" => "world" }
|
@@ -580,7 +580,7 @@ describe Agent do |
||
| 580 | 580 |
@agent.save! |
| 581 | 581 |
|
| 582 | 582 |
@agent.options[:foo] = "bar1" |
| 583 |
- @agent.keep_events_for = 5 |
|
| 583 |
+ @agent.keep_events_for = 5.days |
|
| 584 | 584 |
@agent.save! |
| 585 | 585 |
end |
| 586 | 586 |
end |
@@ -590,7 +590,7 @@ describe Agent do |
||
| 590 | 590 |
time_travel_to @time do |
| 591 | 591 |
expect {
|
| 592 | 592 |
@agent.options[:foo] = "bar1" |
| 593 |
- @agent.keep_events_for = 3 |
|
| 593 |
+ @agent.keep_events_for = 3.days |
|
| 594 | 594 |
@agent.save! |
| 595 | 595 |
}.to change { @event.reload.expires_at }
|
| 596 | 596 |
expect(@event.expires_at.to_i).to be_within(2).of(3.days.from_now.to_i) |
@@ -603,7 +603,7 @@ describe Agent do |
||
| 603 | 603 |
|
| 604 | 604 |
expect {
|
| 605 | 605 |
@agent.options[:foo] = "bar2" |
| 606 |
- @agent.keep_events_for = 3 |
|
| 606 |
+ @agent.keep_events_for = 3.days |
|
| 607 | 607 |
@agent.save! |
| 608 | 608 |
}.to change { @event.reload.expires_at }
|
| 609 | 609 |
expect(@event.expires_at.to_i).to be_within(60 * 61).of(1.days.from_now.to_i) # The larger time is to deal with daylight savings |
@@ -635,7 +635,7 @@ describe Agent do |
||
| 635 | 635 |
@receiver = Agents::CannotBeScheduled.new( |
| 636 | 636 |
name: 'Agent', |
| 637 | 637 |
options: { foo: 'bar3' },
|
| 638 |
- keep_events_for: 3, |
|
| 638 |
+ keep_events_for: 3.days, |
|
| 639 | 639 |
propagate_immediately: true) |
| 640 | 640 |
@receiver.user = users(:bob) |
| 641 | 641 |
@receiver.sources << @sender |
@@ -747,7 +747,7 @@ describe Agent do |
||
| 747 | 747 |
|
| 748 | 748 |
it "sets expires_at on created events" do |
| 749 | 749 |
event = agents(:jane_weather_agent).create_event :payload => { 'hi' => 'there' }
|
| 750 |
- expect(event.expires_at.to_i).to be_within(5).of(agents(:jane_weather_agent).keep_events_for.days.from_now.to_i) |
|
| 750 |
+ expect(event.expires_at.to_i).to be_within(5).of(agents(:jane_weather_agent).keep_events_for.seconds.from_now.to_i) |
|
| 751 | 751 |
end |
| 752 | 752 |
end |
| 753 | 753 |
|
@@ -836,7 +836,7 @@ describe AgentDrop do |
||
| 836 | 836 |
}, |
| 837 | 837 |
}, |
| 838 | 838 |
schedule: 'every_1h', |
| 839 |
- keep_events_for: 2) |
|
| 839 |
+ keep_events_for: 2.days) |
|
| 840 | 840 |
@wsa1.user = users(:bob) |
| 841 | 841 |
@wsa1.save! |
| 842 | 842 |
|
@@ -853,7 +853,7 @@ describe AgentDrop do |
||
| 853 | 853 |
}, |
| 854 | 854 |
}, |
| 855 | 855 |
schedule: 'every_12h', |
| 856 |
- keep_events_for: 2) |
|
| 856 |
+ keep_events_for: 2.days) |
|
| 857 | 857 |
@wsa2.user = users(:bob) |
| 858 | 858 |
@wsa2.save! |
| 859 | 859 |
|
@@ -868,7 +868,7 @@ describe AgentDrop do |
||
| 868 | 868 |
matchers: [], |
| 869 | 869 |
skip_created_at: 'false', |
| 870 | 870 |
}, |
| 871 |
- keep_events_for: 2, |
|
| 871 |
+ keep_events_for: 2.days, |
|
| 872 | 872 |
propagate_immediately: true) |
| 873 | 873 |
@efa.user = users(:bob) |
| 874 | 874 |
@efa.sources << @wsa1 << @wsa2 |
+ 208
- 8
spec/models/agents/data_output_agent_spec.rb
@@ -34,8 +34,18 @@ describe Agents::DataOutputAgent do |
||
| 34 | 34 |
expect(agent).not_to be_valid |
| 35 | 35 |
agent.options[:secrets] = "foo" |
| 36 | 36 |
expect(agent).not_to be_valid |
| 37 |
+ agent.options[:secrets] = "foo/bar" |
|
| 38 |
+ expect(agent).not_to be_valid |
|
| 39 |
+ agent.options[:secrets] = "foo.xml" |
|
| 40 |
+ expect(agent).not_to be_valid |
|
| 41 |
+ agent.options[:secrets] = false |
|
| 42 |
+ expect(agent).not_to be_valid |
|
| 37 | 43 |
agent.options[:secrets] = [] |
| 38 | 44 |
expect(agent).not_to be_valid |
| 45 |
+ agent.options[:secrets] = ["foo.xml"] |
|
| 46 |
+ expect(agent).not_to be_valid |
|
| 47 |
+ agent.options[:secrets] = ["hello", true] |
|
| 48 |
+ expect(agent).not_to be_valid |
|
| 39 | 49 |
agent.options[:secrets] = ["hello"] |
| 40 | 50 |
expect(agent).to be_valid |
| 41 | 51 |
agent.options[:secrets] = ["hello", "world"] |
@@ -83,9 +93,10 @@ describe Agents::DataOutputAgent do |
||
| 83 | 93 |
expect(status).to eq(200) |
| 84 | 94 |
end |
| 85 | 95 |
|
| 86 |
- describe "returning events as RSS and JSON" do |
|
| 96 |
+ describe "outputting events as RSS and JSON" do |
|
| 87 | 97 |
let!(:event1) do |
| 88 | 98 |
agents(:bob_website_agent).create_event :payload => {
|
| 99 |
+ "site_title" => "XKCD", |
|
| 89 | 100 |
"url" => "http://imgs.xkcd.com/comics/evolving.png", |
| 90 | 101 |
"title" => "Evolving", |
| 91 | 102 |
"hovertext" => "Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution." |
@@ -94,6 +105,7 @@ describe Agents::DataOutputAgent do |
||
| 94 | 105 |
|
| 95 | 106 |
let!(:event2) do |
| 96 | 107 |
agents(:bob_website_agent).create_event :payload => {
|
| 108 |
+ "site_title" => "XKCD", |
|
| 97 | 109 |
"url" => "http://imgs.xkcd.com/comics/evolving2.png", |
| 98 | 110 |
"title" => "Evolving again", |
| 99 | 111 |
"date" => '', |
@@ -103,6 +115,7 @@ describe Agents::DataOutputAgent do |
||
| 103 | 115 |
|
| 104 | 116 |
let!(:event3) do |
| 105 | 117 |
agents(:bob_website_agent).create_event :payload => {
|
| 118 |
+ "site_title" => "XKCD", |
|
| 106 | 119 |
"url" => "http://imgs.xkcd.com/comics/evolving0.png", |
| 107 | 120 |
"title" => "Evolving yet again with a past date", |
| 108 | 121 |
"date" => '2014/05/05', |
@@ -117,8 +130,10 @@ describe Agents::DataOutputAgent do |
||
| 117 | 130 |
expect(content_type).to eq('text/xml')
|
| 118 | 131 |
expect(content.gsub(/\s+/, '')).to eq Utils.unindent(<<-XML).gsub(/\s+/, '') |
| 119 | 132 |
<?xml version="1.0" encoding="UTF-8" ?> |
| 120 |
- <rss version="2.0"> |
|
| 133 |
+ <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> |
|
| 121 | 134 |
<channel> |
| 135 |
+ <atom:link href="https://yoursite.com/users/#{agent.user.id}/web_requests/#{agent.id}/secret1.xml" rel="self" type="application/rss+xml"/>
|
|
| 136 |
+ <atom:icon>https://yoursite.com/favicon.ico</atom:icon> |
|
| 122 | 137 |
<title>XKCD comics as a feed</title> |
| 123 | 138 |
<description>This is a feed of recent XKCD comics, generated by Huginn</description> |
| 124 | 139 |
<link>https://yoursite.com</link> |
@@ -131,7 +146,7 @@ describe Agents::DataOutputAgent do |
||
| 131 | 146 |
<description>Secret hovertext: Something else</description> |
| 132 | 147 |
<link>http://imgs.xkcd.com/comics/evolving0.png</link> |
| 133 | 148 |
<pubDate>#{Time.zone.parse(event3.payload['date']).rfc2822}</pubDate>
|
| 134 |
- <guid>#{event3.id}</guid>
|
|
| 149 |
+ <guid isPermaLink="false">#{event3.id}</guid>
|
|
| 135 | 150 |
</item> |
| 136 | 151 |
|
| 137 | 152 |
<item> |
@@ -139,7 +154,7 @@ describe Agents::DataOutputAgent do |
||
| 139 | 154 |
<description>Secret hovertext: Something else</description> |
| 140 | 155 |
<link>http://imgs.xkcd.com/comics/evolving2.png</link> |
| 141 | 156 |
<pubDate>#{event2.created_at.rfc2822}</pubDate>
|
| 142 |
- <guid>#{event2.id}</guid>
|
|
| 157 |
+ <guid isPermaLink="false">#{event2.id}</guid>
|
|
| 143 | 158 |
</item> |
| 144 | 159 |
|
| 145 | 160 |
<item> |
@@ -147,7 +162,7 @@ describe Agents::DataOutputAgent do |
||
| 147 | 162 |
<description>Secret hovertext: Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution.</description> |
| 148 | 163 |
<link>http://imgs.xkcd.com/comics/evolving.png</link> |
| 149 | 164 |
<pubDate>#{event1.created_at.rfc2822}</pubDate>
|
| 150 |
- <guid>#{event1.id}</guid>
|
|
| 165 |
+ <guid isPermaLink="false">#{event1.id}</guid>
|
|
| 151 | 166 |
</item> |
| 152 | 167 |
|
| 153 | 168 |
</channel> |
@@ -170,7 +185,7 @@ describe Agents::DataOutputAgent do |
||
| 170 | 185 |
'title' => 'Evolving yet again with a past date', |
| 171 | 186 |
'description' => 'Secret hovertext: Something else', |
| 172 | 187 |
'link' => 'http://imgs.xkcd.com/comics/evolving0.png', |
| 173 |
- 'guid' => event3.id, |
|
| 188 |
+ 'guid' => {"contents" => event3.id, "isPermaLink" => "false"},
|
|
| 174 | 189 |
'pubDate' => Time.zone.parse(event3.payload['date']).rfc2822, |
| 175 | 190 |
'foo' => 'hi' |
| 176 | 191 |
}, |
@@ -178,7 +193,7 @@ describe Agents::DataOutputAgent do |
||
| 178 | 193 |
'title' => 'Evolving again', |
| 179 | 194 |
'description' => 'Secret hovertext: Something else', |
| 180 | 195 |
'link' => 'http://imgs.xkcd.com/comics/evolving2.png', |
| 181 |
- 'guid' => event2.id, |
|
| 196 |
+ 'guid' => {"contents" => event2.id, "isPermaLink" => "false"},
|
|
| 182 | 197 |
'pubDate' => event2.created_at.rfc2822, |
| 183 | 198 |
'foo' => 'hi' |
| 184 | 199 |
}, |
@@ -186,13 +201,198 @@ describe Agents::DataOutputAgent do |
||
| 186 | 201 |
'title' => 'Evolving', |
| 187 | 202 |
'description' => 'Secret hovertext: Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution.', |
| 188 | 203 |
'link' => 'http://imgs.xkcd.com/comics/evolving.png', |
| 189 |
- 'guid' => event1.id, |
|
| 204 |
+ 'guid' => {"contents" => event1.id, "isPermaLink" => "false"},
|
|
| 190 | 205 |
'pubDate' => event1.created_at.rfc2822, |
| 191 | 206 |
'foo' => 'hi' |
| 192 | 207 |
} |
| 193 | 208 |
] |
| 194 | 209 |
}) |
| 195 | 210 |
end |
| 211 |
+ |
|
| 212 |
+ describe 'ordering' do |
|
| 213 |
+ before do |
|
| 214 |
+ agent.options['events_order'] = ['{{title}}']
|
|
| 215 |
+ end |
|
| 216 |
+ |
|
| 217 |
+ it 'can reorder the events_to_show last events based on a Liquid expression' do |
|
| 218 |
+ asc_content, _status, _content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
|
|
| 219 |
+ expect(asc_content['items'].map {|i| i["title"] }).to eq(["Evolving", "Evolving again", "Evolving yet again with a past date"])
|
|
| 220 |
+ |
|
| 221 |
+ agent.options['events_order'] = [['{{title}}', 'string', true]]
|
|
| 222 |
+ |
|
| 223 |
+ desc_content, _status, _content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
|
|
| 224 |
+ expect(desc_content['items']).to eq(asc_content['items'].reverse) |
|
| 225 |
+ end |
|
| 226 |
+ end |
|
| 227 |
+ |
|
| 228 |
+ describe "interpolating \"events\"" do |
|
| 229 |
+ before do |
|
| 230 |
+ agent.options['template']['title'] = "XKCD comics as a feed{% if events.first.site_title %} ({{events.first.site_title}}){% endif %}"
|
|
| 231 |
+ agent.save! |
|
| 232 |
+ end |
|
| 233 |
+ |
|
| 234 |
+ it "can output RSS" do |
|
| 235 |
+ stub(agent).feed_link { "https://yoursite.com" }
|
|
| 236 |
+ content, status, content_type = agent.receive_web_request({ 'secret' => 'secret1' }, 'get', 'text/xml')
|
|
| 237 |
+ expect(status).to eq(200) |
|
| 238 |
+ expect(content_type).to eq('text/xml')
|
|
| 239 |
+ expect(Nokogiri(content).at('/rss/channel/title/text()').text).to eq('XKCD comics as a feed (XKCD)')
|
|
| 240 |
+ end |
|
| 241 |
+ |
|
| 242 |
+ it "can output JSON" do |
|
| 243 |
+ content, status, content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
|
|
| 244 |
+ expect(status).to eq(200) |
|
| 245 |
+ |
|
| 246 |
+ expect(content['title']).to eq('XKCD comics as a feed (XKCD)')
|
|
| 247 |
+ end |
|
| 248 |
+ end |
|
| 249 |
+ |
|
| 250 |
+ describe "with a specified icon" do |
|
| 251 |
+ before do |
|
| 252 |
+ agent.options['template']['icon'] = 'https://somesite.com/icon.png' |
|
| 253 |
+ agent.save! |
|
| 254 |
+ end |
|
| 255 |
+ |
|
| 256 |
+ it "can output RSS" do |
|
| 257 |
+ stub(agent).feed_link { "https://yoursite.com" }
|
|
| 258 |
+ content, status, content_type = agent.receive_web_request({ 'secret' => 'secret1' }, 'get', 'text/xml')
|
|
| 259 |
+ expect(status).to eq(200) |
|
| 260 |
+ expect(content_type).to eq('text/xml')
|
|
| 261 |
+ expect(Nokogiri(content).at('/rss/channel/atom:icon/text()').text).to eq('https://somesite.com/icon.png')
|
|
| 262 |
+ end |
|
| 263 |
+ end |
|
| 264 |
+ end |
|
| 265 |
+ |
|
| 266 |
+ describe "outputting nesting" do |
|
| 267 |
+ before do |
|
| 268 |
+ agent.options['template']['item']['enclosure'] = {
|
|
| 269 |
+ "_attributes" => {
|
|
| 270 |
+ "type" => "audio/mpeg", |
|
| 271 |
+ "url" => "{{media_url}}"
|
|
| 272 |
+ } |
|
| 273 |
+ } |
|
| 274 |
+ agent.options['template']['item']['foo'] = {
|
|
| 275 |
+ "_attributes" => {
|
|
| 276 |
+ "attr" => "attr-value-{{foo}}"
|
|
| 277 |
+ }, |
|
| 278 |
+ "_contents" => "Foo: {{foo}}"
|
|
| 279 |
+ } |
|
| 280 |
+ agent.options['template']['item']['nested'] = {
|
|
| 281 |
+ "_attributes" => {
|
|
| 282 |
+ "key" => "value" |
|
| 283 |
+ }, |
|
| 284 |
+ "_contents" => {
|
|
| 285 |
+ "title" => "some title" |
|
| 286 |
+ } |
|
| 287 |
+ } |
|
| 288 |
+ agent.options['template']['item']['simpleNested'] = {
|
|
| 289 |
+ "title" => "some title", |
|
| 290 |
+ "complex" => {
|
|
| 291 |
+ "_attributes" => {
|
|
| 292 |
+ "key" => "value" |
|
| 293 |
+ }, |
|
| 294 |
+ "_contents" => {
|
|
| 295 |
+ "first" => {
|
|
| 296 |
+ "_attributes" => {
|
|
| 297 |
+ "a" => "b" |
|
| 298 |
+ }, |
|
| 299 |
+ "_contents" => {
|
|
| 300 |
+ "second" => "value" |
|
| 301 |
+ } |
|
| 302 |
+ } |
|
| 303 |
+ } |
|
| 304 |
+ } |
|
| 305 |
+ } |
|
| 306 |
+ agent.save! |
|
| 307 |
+ end |
|
| 308 |
+ |
|
| 309 |
+ let!(:event) do |
|
| 310 |
+ agents(:bob_website_agent).create_event :payload => {
|
|
| 311 |
+ "url" => "http://imgs.xkcd.com/comics/evolving.png", |
|
| 312 |
+ "title" => "Evolving", |
|
| 313 |
+ "hovertext" => "Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution.", |
|
| 314 |
+ "media_url" => "http://google.com/audio.mpeg", |
|
| 315 |
+ "foo" => 1 |
|
| 316 |
+ } |
|
| 317 |
+ end |
|
| 318 |
+ |
|
| 319 |
+ it "can output JSON" do |
|
| 320 |
+ content, status, content_type = agent.receive_web_request({ 'secret' => 'secret2' }, 'get', 'application/json')
|
|
| 321 |
+ expect(status).to eq(200) |
|
| 322 |
+ expect(content['items'].first).to eq( |
|
| 323 |
+ {
|
|
| 324 |
+ 'title' => 'Evolving', |
|
| 325 |
+ 'description' => 'Secret hovertext: Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution.', |
|
| 326 |
+ 'link' => 'http://imgs.xkcd.com/comics/evolving.png', |
|
| 327 |
+ 'guid' => {"contents" => event.id, "isPermaLink" => "false"},
|
|
| 328 |
+ 'pubDate' => event.created_at.rfc2822, |
|
| 329 |
+ 'enclosure' => {
|
|
| 330 |
+ "type" => "audio/mpeg", |
|
| 331 |
+ "url" => "http://google.com/audio.mpeg" |
|
| 332 |
+ }, |
|
| 333 |
+ 'foo' => {
|
|
| 334 |
+ 'attr' => 'attr-value-1', |
|
| 335 |
+ 'contents' => 'Foo: 1' |
|
| 336 |
+ }, |
|
| 337 |
+ 'nested' => {
|
|
| 338 |
+ "key" => "value", |
|
| 339 |
+ "title" => "some title" |
|
| 340 |
+ }, |
|
| 341 |
+ 'simpleNested' => {
|
|
| 342 |
+ "title" => "some title", |
|
| 343 |
+ "complex" => {
|
|
| 344 |
+ "key"=>"value", |
|
| 345 |
+ "first" => {
|
|
| 346 |
+ "a" => "b", |
|
| 347 |
+ "second"=>"value" |
|
| 348 |
+ } |
|
| 349 |
+ } |
|
| 350 |
+ } |
|
| 351 |
+ } |
|
| 352 |
+ ) |
|
| 353 |
+ end |
|
| 354 |
+ |
|
| 355 |
+ it "can output RSS" do |
|
| 356 |
+ stub(agent).feed_link { "https://yoursite.com" }
|
|
| 357 |
+ content, status, content_type = agent.receive_web_request({ 'secret' => 'secret1' }, 'get', 'text/xml')
|
|
| 358 |
+ expect(status).to eq(200) |
|
| 359 |
+ expect(content_type).to eq('text/xml')
|
|
| 360 |
+ expect(content.gsub(/\s+/, '')).to eq Utils.unindent(<<-XML).gsub(/\s+/, '') |
|
| 361 |
+ <?xml version="1.0" encoding="UTF-8" ?> |
|
| 362 |
+ <rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"> |
|
| 363 |
+ <channel> |
|
| 364 |
+ <atom:link href="https://yoursite.com/users/#{agent.user.id}/web_requests/#{agent.id}/secret1.xml" rel="self" type="application/rss+xml"/>
|
|
| 365 |
+ <atom:icon>https://yoursite.com/favicon.ico</atom:icon> |
|
| 366 |
+ <title>XKCD comics as a feed</title> |
|
| 367 |
+ <description>This is a feed of recent XKCD comics, generated by Huginn</description> |
|
| 368 |
+ <link>https://yoursite.com</link> |
|
| 369 |
+ <lastBuildDate>#{Time.now.rfc2822}</lastBuildDate>
|
|
| 370 |
+ <pubDate>#{Time.now.rfc2822}</pubDate>
|
|
| 371 |
+ <ttl>60</ttl> |
|
| 372 |
+ |
|
| 373 |
+ <item> |
|
| 374 |
+ <title>Evolving</title> |
|
| 375 |
+ <description>Secret hovertext: Biologists play reverse Pokemon, trying to avoid putting any one team member on the front lines long enough for the experience to cause evolution.</description> |
|
| 376 |
+ <link>http://imgs.xkcd.com/comics/evolving.png</link> |
|
| 377 |
+ <pubDate>#{event.created_at.rfc2822}</pubDate>
|
|
| 378 |
+ <enclosure type="audio/mpeg" url="http://google.com/audio.mpeg" /> |
|
| 379 |
+ <foo attr="attr-value-1">Foo: 1</foo> |
|
| 380 |
+ <nested key="value"><title>some title</title></nested> |
|
| 381 |
+ <simpleNested> |
|
| 382 |
+ <title>some title</title> |
|
| 383 |
+ <complex key="value"> |
|
| 384 |
+ <first a="b"> |
|
| 385 |
+ <second>value</second> |
|
| 386 |
+ </first> |
|
| 387 |
+ </complex> |
|
| 388 |
+ </simpleNested> |
|
| 389 |
+ <guid isPermaLink="false">#{event.id}</guid>
|
|
| 390 |
+ </item> |
|
| 391 |
+ |
|
| 392 |
+ </channel> |
|
| 393 |
+ </rss> |
|
| 394 |
+ XML |
|
| 395 |
+ end |
|
| 196 | 396 |
end |
| 197 | 397 |
end |
| 198 | 398 |
end |
+ 22
- 0
spec/models/agents/de_duplication_agent_spec.rb
@@ -40,6 +40,19 @@ describe Agents::DeDuplicationAgent do |
||
| 40 | 40 |
end |
| 41 | 41 |
end |
| 42 | 42 |
|
| 43 |
+ describe '#initialize_memory' do |
|
| 44 |
+ it 'sets properties to an empty array' do |
|
| 45 |
+ expect(@checker.memory['properties']).to eq([]) |
|
| 46 |
+ end |
|
| 47 |
+ |
|
| 48 |
+ it 'does not override an existing value' do |
|
| 49 |
+ @checker.memory['properties'] = [1,2,3] |
|
| 50 |
+ @checker.save |
|
| 51 |
+ @checker.reload |
|
| 52 |
+ expect(@checker.memory['properties']).to eq([1,2,3]) |
|
| 53 |
+ end |
|
| 54 |
+ end |
|
| 55 |
+ |
|
| 43 | 56 |
describe "#working?" do |
| 44 | 57 |
before :each do |
| 45 | 58 |
# Need to create an event otherwise event_created_within? returns nil |
@@ -123,5 +136,14 @@ describe Agents::DeDuplicationAgent do |
||
| 123 | 136 |
}.to change(Event, :count).by(1) |
| 124 | 137 |
expect(@checker.memory['properties'].last).to eq('3023526198')
|
| 125 | 138 |
end |
| 139 |
+ |
|
| 140 |
+ it "should still work after the memory was cleared" do |
|
| 141 |
+ @checker.memory = {}
|
|
| 142 |
+ @checker.save |
|
| 143 |
+ @checker.reload |
|
| 144 |
+ expect {
|
|
| 145 |
+ @checker.receive([@event]) |
|
| 146 |
+ }.not_to raise_error |
|
| 147 |
+ end |
|
| 126 | 148 |
end |
| 127 | 149 |
end |
+ 1
- 1
spec/models/agents/ftpsite_agent_spec.rb
@@ -9,7 +9,7 @@ describe Agents::FtpsiteAgent do |
||
| 9 | 9 |
'url' => "ftp://ftp.example.org/pub/releases/", |
| 10 | 10 |
'patterns' => ["example*.tar.gz"], |
| 11 | 11 |
} |
| 12 |
- @checker = Agents::FtpsiteAgent.new(:name => "Example", :options => @site, :keep_events_for => 2) |
|
| 12 |
+ @checker = Agents::FtpsiteAgent.new(:name => "Example", :options => @site, :keep_events_for => 2.days) |
|
| 13 | 13 |
@checker.user = users(:bob) |
| 14 | 14 |
@checker.save! |
| 15 | 15 |
end |
+ 1
- 1
spec/models/agents/imap_folder_agent_spec.rb
@@ -14,7 +14,7 @@ describe Agents::ImapFolderAgent do |
||
| 14 | 14 |
'conditions' => {
|
| 15 | 15 |
} |
| 16 | 16 |
} |
| 17 |
- @checker = Agents::ImapFolderAgent.new(:name => 'Example', :options => @site, :keep_events_for => 2) |
|
| 17 |
+ @checker = Agents::ImapFolderAgent.new(:name => 'Example', :options => @site, :keep_events_for => 2.days) |
|
| 18 | 18 |
@checker.user = users(:bob) |
| 19 | 19 |
@checker.save! |
| 20 | 20 |
|
+ 65
- 3
spec/models/agents/rss_agent_spec.rb
@@ -25,6 +25,9 @@ describe Agents::RssAgent do |
||
| 25 | 25 |
agent.options['url'] = "http://google.com" |
| 26 | 26 |
expect(agent).to be_valid |
| 27 | 27 |
|
| 28 |
+ agent.options['url'] = ["http://google.com", "http://yahoo.com"] |
|
| 29 |
+ expect(agent).to be_valid |
|
| 30 |
+ |
|
| 28 | 31 |
agent.options['url'] = "" |
| 29 | 32 |
expect(agent).not_to be_valid |
| 30 | 33 |
|
@@ -56,9 +59,26 @@ describe Agents::RssAgent do |
||
| 56 | 59 |
agent.check |
| 57 | 60 |
}.to change { agent.events.count }.by(20)
|
| 58 | 61 |
|
| 59 |
- event = agent.events.last |
|
| 60 |
- expect(event.payload['url']).to eq("https://github.com/cantino/huginn/commit/d0a844662846cf3c83b94c637c1803f03db5a5b0")
|
|
| 61 |
- expect(event.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/d0a844662846cf3c83b94c637c1803f03db5a5b0"]) |
|
| 62 |
+ first, *, last = agent.events.last(20) |
|
| 63 |
+ expect(first.payload['url']).to eq("https://github.com/cantino/huginn/commit/d0a844662846cf3c83b94c637c1803f03db5a5b0")
|
|
| 64 |
+ expect(first.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/d0a844662846cf3c83b94c637c1803f03db5a5b0"]) |
|
| 65 |
+ expect(last.payload['url']).to eq("https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af")
|
|
| 66 |
+ expect(last.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af"]) |
|
| 67 |
+ end |
|
| 68 |
+ |
|
| 69 |
+ it "should emit items as events in the order specified in the events_order option" do |
|
| 70 |
+ expect {
|
|
| 71 |
+ agent.options['events_order'] = ['{{title | replace_regex: "^[[:space:]]+", "" }}']
|
|
| 72 |
+ agent.check |
|
| 73 |
+ }.to change { agent.events.count }.by(20)
|
|
| 74 |
+ |
|
| 75 |
+ first, *, last = agent.events.last(20) |
|
| 76 |
+ expect(first.payload['title'].strip).to eq('upgrade rails and gems')
|
|
| 77 |
+ expect(first.payload['url']).to eq("https://github.com/cantino/huginn/commit/87a7abda23a82305d7050ac0bb400ce36c863d01")
|
|
| 78 |
+ expect(first.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/87a7abda23a82305d7050ac0bb400ce36c863d01"]) |
|
| 79 |
+ expect(last.payload['title'].strip).to eq('Dashed line in a diagram indicates propagate_immediately being false.')
|
|
| 80 |
+ expect(last.payload['url']).to eq("https://github.com/cantino/huginn/commit/0e80f5341587aace2c023b06eb9265b776ac4535")
|
|
| 81 |
+ expect(last.payload['urls']).to eq(["https://github.com/cantino/huginn/commit/0e80f5341587aace2c023b06eb9265b776ac4535"]) |
|
| 62 | 82 |
end |
| 63 | 83 |
|
| 64 | 84 |
it "should track ids and not re-emit the same item when seen again" do |
@@ -82,6 +102,38 @@ describe Agents::RssAgent do |
||
| 82 | 102 |
agent.check |
| 83 | 103 |
expect(agent.memory['seen_ids'].length).to eq(500) |
| 84 | 104 |
end |
| 105 |
+ |
|
| 106 |
+ it "should support an array of URLs" do |
|
| 107 |
+ agent.options['url'] = ["https://github.com/cantino/huginn/commits/master.atom", "http://feeds.feedburner.com/SlickdealsnetFP?format=atom"] |
|
| 108 |
+ agent.save! |
|
| 109 |
+ |
|
| 110 |
+ expect {
|
|
| 111 |
+ agent.check |
|
| 112 |
+ }.to change { agent.events.count }.by(20 + 79)
|
|
| 113 |
+ end |
|
| 114 |
+ |
|
| 115 |
+ it "should fetch one event per run" do |
|
| 116 |
+ agent.options['url'] = ["https://github.com/cantino/huginn/commits/master.atom"] |
|
| 117 |
+ |
|
| 118 |
+ agent.options['max_events_per_run'] = 1 |
|
| 119 |
+ agent.check |
|
| 120 |
+ expect(agent.events.count).to eq(1) |
|
| 121 |
+ end |
|
| 122 |
+ |
|
| 123 |
+ it "should fetch all events per run" do |
|
| 124 |
+ agent.options['url'] = ["https://github.com/cantino/huginn/commits/master.atom"] |
|
| 125 |
+ |
|
| 126 |
+ # <= 0 should ignore option and get all |
|
| 127 |
+ agent.options['max_events_per_run'] = 0 |
|
| 128 |
+ agent.check |
|
| 129 |
+ expect(agent.events.count).to eq(20) |
|
| 130 |
+ |
|
| 131 |
+ agent.options['max_events_per_run'] = -1 |
|
| 132 |
+ expect {
|
|
| 133 |
+ agent.check |
|
| 134 |
+ }.to_not change { agent.events.count }
|
|
| 135 |
+ end |
|
| 136 |
+ |
|
| 85 | 137 |
end |
| 86 | 138 |
|
| 87 | 139 |
context "when no ids are available" do |
@@ -96,4 +148,14 @@ describe Agents::RssAgent do |
||
| 96 | 148 |
expect(agent.memory['seen_ids']).to eq(agent.events.map {|e| Digest::MD5.hexdigest(e.payload['content']) })
|
| 97 | 149 |
end |
| 98 | 150 |
end |
| 151 |
+ |
|
| 152 |
+ describe 'logging errors with the feed url' do |
|
| 153 |
+ it 'includes the feed URL when an exception is raised' do |
|
| 154 |
+ mock(FeedNormalizer::FeedNormalizer).parse(anything) { raise StandardError.new("Some error!") }
|
|
| 155 |
+ expect(lambda {
|
|
| 156 |
+ agent.check |
|
| 157 |
+ }).not_to raise_error |
|
| 158 |
+ expect(agent.logs.last.message).to match(%r[Failed to fetch https://github.com]) |
|
| 159 |
+ end |
|
| 160 |
+ end |
|
| 99 | 161 |
end |
+ 2
- 0
spec/models/agents/twitter_user_agent_spec.rb
@@ -7,6 +7,8 @@ describe Agents::TwitterUserAgent do |
||
| 7 | 7 |
|
| 8 | 8 |
@opts = {
|
| 9 | 9 |
:username => "tectonic", |
| 10 |
+ :include_retweets => "true", |
|
| 11 |
+ :exclude_replies => "false", |
|
| 10 | 12 |
:expected_update_period_in_days => "2", |
| 11 | 13 |
:starting_at => "Jan 01 00:00:01 +0000 2000", |
| 12 | 14 |
:consumer_key => "---", |
+ 134
- 5
spec/models/agents/website_agent_spec.rb
@@ -20,7 +20,7 @@ describe Agents::WebsiteAgent do |
||
| 20 | 20 |
'hovertext' => { 'css' => "#comic img", 'value' => "@title" }
|
| 21 | 21 |
} |
| 22 | 22 |
} |
| 23 |
- @checker = Agents::WebsiteAgent.new(:name => "xkcd", :options => @valid_options, :keep_events_for => 2) |
|
| 23 |
+ @checker = Agents::WebsiteAgent.new(:name => "xkcd", :options => @valid_options, :keep_events_for => 2.days) |
|
| 24 | 24 |
@checker.user = users(:bob) |
| 25 | 25 |
@checker.save! |
| 26 | 26 |
end |
@@ -75,6 +75,23 @@ describe Agents::WebsiteAgent do |
||
| 75 | 75 |
@checker.options['force_encoding'] = 'UTF-42' |
| 76 | 76 |
expect(@checker).not_to be_valid |
| 77 | 77 |
end |
| 78 |
+ |
|
| 79 |
+ context "in 'json' type" do |
|
| 80 |
+ it "should ensure that all extractions have a 'path'" do |
|
| 81 |
+ @checker.options['type'] = 'json' |
|
| 82 |
+ @checker.options['extract'] = {
|
|
| 83 |
+ 'url' => { 'foo' => 'bar' },
|
|
| 84 |
+ } |
|
| 85 |
+ expect(@checker).to_not be_valid |
|
| 86 |
+ expect(@checker.errors_on(:base)).to include(/When type is json, all extractions must have a path attribute/) |
|
| 87 |
+ |
|
| 88 |
+ @checker.options['type'] = 'json' |
|
| 89 |
+ @checker.options['extract'] = {
|
|
| 90 |
+ 'url' => { 'path' => 'bar' },
|
|
| 91 |
+ } |
|
| 92 |
+ expect(@checker).to be_valid |
|
| 93 |
+ end |
|
| 94 |
+ end |
|
| 78 | 95 |
end |
| 79 | 96 |
|
| 80 | 97 |
describe "#check" do |
@@ -179,7 +196,6 @@ describe Agents::WebsiteAgent do |
||
| 179 | 196 |
|
| 180 | 197 |
checker.check |
| 181 | 198 |
event = Event.last |
| 182 |
- puts event.payload |
|
| 183 | 199 |
expect(event.payload['version']).to eq(2) |
| 184 | 200 |
end |
| 185 | 201 |
end |
@@ -352,6 +368,108 @@ describe Agents::WebsiteAgent do |
||
| 352 | 368 |
expect(event.payload['response_info']).to eq('The reponse was 200 OK.')
|
| 353 | 369 |
end |
| 354 | 370 |
|
| 371 |
+ describe "XML" do |
|
| 372 |
+ before do |
|
| 373 |
+ stub_request(:any, /github_rss/).to_return( |
|
| 374 |
+ body: File.read(Rails.root.join("spec/data_fixtures/github_rss.atom")),
|
|
| 375 |
+ status: 200 |
|
| 376 |
+ ) |
|
| 377 |
+ |
|
| 378 |
+ @checker = Agents::WebsiteAgent.new(name: 'github', options: {
|
|
| 379 |
+ 'name' => 'GitHub', |
|
| 380 |
+ 'expected_update_period_in_days' => '2', |
|
| 381 |
+ 'type' => 'xml', |
|
| 382 |
+ 'url' => 'http://example.com/github_rss.atom', |
|
| 383 |
+ 'mode' => 'on_change', |
|
| 384 |
+ 'extract' => {
|
|
| 385 |
+ 'title' => { 'xpath' => '/feed/entry', 'value' => 'normalize-space(./title)' },
|
|
| 386 |
+ 'url' => { 'xpath' => '/feed/entry', 'value' => './link[1]/@href' },
|
|
| 387 |
+ 'thumbnail' => { 'xpath' => '/feed/entry', 'value' => './thumbnail/@url' },
|
|
| 388 |
+ } |
|
| 389 |
+ }, keep_events_for: 2.days) |
|
| 390 |
+ @checker.user = users(:bob) |
|
| 391 |
+ @checker.save! |
|
| 392 |
+ end |
|
| 393 |
+ |
|
| 394 |
+ it "works with XPath" do |
|
| 395 |
+ expect {
|
|
| 396 |
+ @checker.check |
|
| 397 |
+ }.to change { Event.count }.by(20)
|
|
| 398 |
+ event = Event.last |
|
| 399 |
+ expect(event.payload['title']).to eq('Shift to dev group')
|
|
| 400 |
+ expect(event.payload['url']).to eq('https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af')
|
|
| 401 |
+ expect(event.payload['thumbnail']).to eq('https://avatars3.githubusercontent.com/u/365751?s=30')
|
|
| 402 |
+ end |
|
| 403 |
+ |
|
| 404 |
+ it "works with XPath with namespaces unstripped" do |
|
| 405 |
+ @checker.options['use_namespaces'] = 'true' |
|
| 406 |
+ @checker.save! |
|
| 407 |
+ expect {
|
|
| 408 |
+ @checker.check |
|
| 409 |
+ }.to change { Event.count }.by(0)
|
|
| 410 |
+ |
|
| 411 |
+ @checker.options['extract'] = {
|
|
| 412 |
+ 'title' => { 'xpath' => '/xmlns:feed/xmlns:entry', 'value' => 'normalize-space(./xmlns:title)' },
|
|
| 413 |
+ 'url' => { 'xpath' => '/xmlns:feed/xmlns:entry', 'value' => './xmlns:link[1]/@href' },
|
|
| 414 |
+ 'thumbnail' => { 'xpath' => '/xmlns:feed/xmlns:entry', 'value' => './media:thumbnail/@url' },
|
|
| 415 |
+ } |
|
| 416 |
+ @checker.save! |
|
| 417 |
+ expect {
|
|
| 418 |
+ @checker.check |
|
| 419 |
+ }.to change { Event.count }.by(20)
|
|
| 420 |
+ event = Event.last |
|
| 421 |
+ expect(event.payload['title']).to eq('Shift to dev group')
|
|
| 422 |
+ expect(event.payload['url']).to eq('https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af')
|
|
| 423 |
+ expect(event.payload['thumbnail']).to eq('https://avatars3.githubusercontent.com/u/365751?s=30')
|
|
| 424 |
+ end |
|
| 425 |
+ |
|
| 426 |
+ it "works with CSS selectors" do |
|
| 427 |
+ @checker.options['extract'] = {
|
|
| 428 |
+ 'title' => { 'css' => 'feed > entry', 'value' => 'normalize-space(./title)' },
|
|
| 429 |
+ 'url' => { 'css' => 'feed > entry', 'value' => './link[1]/@href' },
|
|
| 430 |
+ 'thumbnail' => { 'css' => 'feed > entry', 'value' => './thumbnail/@url' },
|
|
| 431 |
+ } |
|
| 432 |
+ @checker.save! |
|
| 433 |
+ expect {
|
|
| 434 |
+ @checker.check |
|
| 435 |
+ }.to change { Event.count }.by(20)
|
|
| 436 |
+ event = Event.last |
|
| 437 |
+ expect(event.payload['title']).to be_empty |
|
| 438 |
+ expect(event.payload['thumbnail']).to be_empty |
|
| 439 |
+ |
|
| 440 |
+ @checker.options['extract'] = {
|
|
| 441 |
+ 'title' => { 'css' => 'feed > entry', 'value' => 'normalize-space(./xmlns:title)' },
|
|
| 442 |
+ 'url' => { 'css' => 'feed > entry', 'value' => './xmlns:link[1]/@href' },
|
|
| 443 |
+ 'thumbnail' => { 'css' => 'feed > entry', 'value' => './media:thumbnail/@url' },
|
|
| 444 |
+ } |
|
| 445 |
+ @checker.save! |
|
| 446 |
+ expect {
|
|
| 447 |
+ @checker.check |
|
| 448 |
+ }.to change { Event.count }.by(20)
|
|
| 449 |
+ event = Event.last |
|
| 450 |
+ expect(event.payload['title']).to eq('Shift to dev group')
|
|
| 451 |
+ expect(event.payload['url']).to eq('https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af')
|
|
| 452 |
+ expect(event.payload['thumbnail']).to eq('https://avatars3.githubusercontent.com/u/365751?s=30')
|
|
| 453 |
+ end |
|
| 454 |
+ |
|
| 455 |
+ it "works with CSS selectors with namespaces stripped" do |
|
| 456 |
+ @checker.options['extract'] = {
|
|
| 457 |
+ 'title' => { 'css' => 'feed > entry', 'value' => 'normalize-space(./title)' },
|
|
| 458 |
+ 'url' => { 'css' => 'feed > entry', 'value' => './link[1]/@href' },
|
|
| 459 |
+ 'thumbnail' => { 'css' => 'feed > entry', 'value' => './thumbnail/@url' },
|
|
| 460 |
+ } |
|
| 461 |
+ @checker.options['use_namespaces'] = 'false' |
|
| 462 |
+ @checker.save! |
|
| 463 |
+ expect {
|
|
| 464 |
+ @checker.check |
|
| 465 |
+ }.to change { Event.count }.by(20)
|
|
| 466 |
+ event = Event.last |
|
| 467 |
+ expect(event.payload['title']).to eq('Shift to dev group')
|
|
| 468 |
+ expect(event.payload['url']).to eq('https://github.com/cantino/huginn/commit/d465158f77dcd9078697e6167b50abbfdfa8b1af')
|
|
| 469 |
+ expect(event.payload['thumbnail']).to eq('https://avatars3.githubusercontent.com/u/365751?s=30')
|
|
| 470 |
+ end |
|
| 471 |
+ end |
|
| 472 |
+ |
|
| 355 | 473 |
describe "JSON" do |
| 356 | 474 |
it "works with paths" do |
| 357 | 475 |
json = {
|
@@ -459,7 +577,7 @@ fire: hot |
||
| 459 | 577 |
'mode' => 'on_change', |
| 460 | 578 |
'extract' => {
|
| 461 | 579 |
'word' => { 'regexp' => '^(.+?): (.+)$', index: 1 },
|
| 462 |
- 'property' => { 'regexp' => '^(.+?): (.+)$', index: 2 },
|
|
| 580 |
+ 'property' => { 'regexp' => '^(.+?): (.+)$', index: '2' },
|
|
| 463 | 581 |
} |
| 464 | 582 |
} |
| 465 | 583 |
@checker = Agents::WebsiteAgent.new(name: 'Text Site', options: site) |
@@ -467,7 +585,7 @@ fire: hot |
||
| 467 | 585 |
@checker.save! |
| 468 | 586 |
end |
| 469 | 587 |
|
| 470 |
- it "works with regexp" do |
|
| 588 |
+ it "works with regexp with named capture" do |
|
| 471 | 589 |
@checker.options = @checker.options.merge('extract' => {
|
| 472 | 590 |
'word' => { 'regexp' => '^(?<word>.+?): (?<property>.+)$', index: 'word' },
|
| 473 | 591 |
'property' => { 'regexp' => '^(?<word>.+?): (?<property>.+)$', index: 'property' },
|
@@ -484,7 +602,7 @@ fire: hot |
||
| 484 | 602 |
expect(event2.payload['property']).to eq('hot')
|
| 485 | 603 |
end |
| 486 | 604 |
|
| 487 |
- it "works with regexp with named capture" do |
|
| 605 |
+ it "works with regexp" do |
|
| 488 | 606 |
expect {
|
| 489 | 607 |
@checker.check |
| 490 | 608 |
}.to change { Event.count }.by(2)
|
@@ -515,6 +633,17 @@ fire: hot |
||
| 515 | 633 |
}.to change { Event.count }.by(1)
|
| 516 | 634 |
end |
| 517 | 635 |
|
| 636 |
+ it "should use url_from_event as url to scrape if it exists when receiving an event" do |
|
| 637 |
+ stub = stub_request(:any, 'http://example.org/?url=http%3A%2F%2Fxkcd.com') |
|
| 638 |
+ |
|
| 639 |
+ @checker.options = @valid_options.merge( |
|
| 640 |
+ 'url_from_event' => 'http://example.org/?url={{url | uri_escape}}'
|
|
| 641 |
+ ) |
|
| 642 |
+ @checker.receive([@event]) |
|
| 643 |
+ |
|
| 644 |
+ expect(stub).to have_been_requested |
|
| 645 |
+ end |
|
| 646 |
+ |
|
| 518 | 647 |
it "should interpolate values from incoming event payload" do |
| 519 | 648 |
expect {
|
| 520 | 649 |
@valid_options['extract'] = {
|
+ 2
- 1
spec/models/agents/wunderlist_agent_spec.rb
@@ -54,8 +54,9 @@ describe Agents::WunderlistAgent do |
||
| 54 | 54 |
|
| 55 | 55 |
describe "#receive" do |
| 56 | 56 |
it "send a message to the hipchat" do |
| 57 |
- stub_request(:post, 'https://a.wunderlist.com/api/v1/tasks').with { |request| request.body == 'abc'}
|
|
| 57 |
+ stub_request(:post, 'https://a.wunderlist.com/api/v1/tasks') |
|
| 58 | 58 |
@checker.receive([@event]) |
| 59 |
+ expect(WebMock).to have_requested(:post, "https://a.wunderlist.com/api/v1/tasks") |
|
| 59 | 60 |
end |
| 60 | 61 |
end |
| 61 | 62 |
|
+ 89
- 9
spec/models/scenario_import_spec.rb
@@ -30,7 +30,7 @@ describe ScenarioImport do |
||
| 30 | 30 |
:type => "Agents::WeatherAgent", |
| 31 | 31 |
:name => "a weather agent", |
| 32 | 32 |
:schedule => "5pm", |
| 33 |
- :keep_events_for => 14, |
|
| 33 |
+ :keep_events_for => 14.days, |
|
| 34 | 34 |
:disabled => true, |
| 35 | 35 |
:guid => "a-weather-agent", |
| 36 | 36 |
:options => weather_agent_options |
@@ -61,6 +61,7 @@ describe ScenarioImport do |
||
| 61 | 61 |
end |
| 62 | 62 |
let(:valid_parsed_data) do |
| 63 | 63 |
{
|
| 64 |
+ :schema_version => 1, |
|
| 64 | 65 |
:name => name, |
| 65 | 66 |
:description => description, |
| 66 | 67 |
:guid => guid, |
@@ -74,7 +75,8 @@ describe ScenarioImport do |
||
| 74 | 75 |
], |
| 75 | 76 |
:links => [ |
| 76 | 77 |
{ :source => 0, :receiver => 1 }
|
| 77 |
- ] |
|
| 78 |
+ ], |
|
| 79 |
+ :control_links => [] |
|
| 78 | 80 |
} |
| 79 | 81 |
end |
| 80 | 82 |
let(:valid_data) { valid_parsed_data.to_json }
|
@@ -203,7 +205,7 @@ describe ScenarioImport do |
||
| 203 | 205 |
|
| 204 | 206 |
expect(weather_agent.name).to eq("a weather agent")
|
| 205 | 207 |
expect(weather_agent.schedule).to eq("5pm")
|
| 206 |
- expect(weather_agent.keep_events_for).to eq(14) |
|
| 208 |
+ expect(weather_agent.keep_events_for).to eq(14.days) |
|
| 207 | 209 |
expect(weather_agent.propagate_immediately).to be_falsey |
| 208 | 210 |
expect(weather_agent).to be_disabled |
| 209 | 211 |
expect(weather_agent.memory).to be_empty |
@@ -226,6 +228,55 @@ describe ScenarioImport do |
||
| 226 | 228 |
scenario_import.import |
| 227 | 229 |
}.to change { users(:bob).agents.count }.by(2)
|
| 228 | 230 |
end |
| 231 |
+ |
|
| 232 |
+ context "when the schema_version is less than 1" do |
|
| 233 |
+ before do |
|
| 234 |
+ valid_parsed_weather_agent_data[:keep_events_for] = 2 |
|
| 235 |
+ valid_parsed_data.delete(:schema_version) |
|
| 236 |
+ end |
|
| 237 |
+ |
|
| 238 |
+ it "translates keep_events_for from days to seconds" do |
|
| 239 |
+ scenario_import.import |
|
| 240 |
+ expect(scenario_import.errors).to be_empty |
|
| 241 |
+ weather_agent = scenario_import.scenario.agents.find_by(:guid => "a-weather-agent") |
|
| 242 |
+ trigger_agent = scenario_import.scenario.agents.find_by(:guid => "a-trigger-agent") |
|
| 243 |
+ |
|
| 244 |
+ expect(weather_agent.keep_events_for).to eq(2.days) |
|
| 245 |
+ expect(trigger_agent.keep_events_for).to eq(0) |
|
| 246 |
+ end |
|
| 247 |
+ end |
|
| 248 |
+ |
|
| 249 |
+ describe "with control links" do |
|
| 250 |
+ it 'creates the links' do |
|
| 251 |
+ valid_parsed_data[:control_links] = [ |
|
| 252 |
+ { :controller => 1, :control_target => 0 }
|
|
| 253 |
+ ] |
|
| 254 |
+ |
|
| 255 |
+ expect {
|
|
| 256 |
+ scenario_import.import |
|
| 257 |
+ }.to change { users(:bob).agents.count }.by(2)
|
|
| 258 |
+ |
|
| 259 |
+ weather_agent = scenario_import.scenario.agents.find_by(:guid => "a-weather-agent") |
|
| 260 |
+ trigger_agent = scenario_import.scenario.agents.find_by(:guid => "a-trigger-agent") |
|
| 261 |
+ |
|
| 262 |
+ expect(trigger_agent.sources).to eq([weather_agent]) |
|
| 263 |
+ expect(weather_agent.controllers.to_a).to eq([trigger_agent]) |
|
| 264 |
+ expect(trigger_agent.control_targets.to_a).to eq([weather_agent]) |
|
| 265 |
+ end |
|
| 266 |
+ |
|
| 267 |
+ it "doesn't crash without any control links" do |
|
| 268 |
+ valid_parsed_data.delete(:control_links) |
|
| 269 |
+ |
|
| 270 |
+ expect {
|
|
| 271 |
+ scenario_import.import |
|
| 272 |
+ }.to change { users(:bob).agents.count }.by(2)
|
|
| 273 |
+ |
|
| 274 |
+ weather_agent = scenario_import.scenario.agents.find_by(:guid => "a-weather-agent") |
|
| 275 |
+ trigger_agent = scenario_import.scenario.agents.find_by(:guid => "a-trigger-agent") |
|
| 276 |
+ |
|
| 277 |
+ expect(trigger_agent.sources).to eq([weather_agent]) |
|
| 278 |
+ end |
|
| 279 |
+ end |
|
| 229 | 280 |
end |
| 230 | 281 |
|
| 231 | 282 |
describe "#generate_diff" do |
@@ -309,7 +360,7 @@ describe ScenarioImport do |
||
| 309 | 360 |
|
| 310 | 361 |
expect(weather_agent.name).to eq("a weather agent")
|
| 311 | 362 |
expect(weather_agent.schedule).to eq("5pm")
|
| 312 |
- expect(weather_agent.keep_events_for).to eq(14) |
|
| 363 |
+ expect(weather_agent.keep_events_for).to eq(14.days) |
|
| 313 | 364 |
expect(weather_agent.propagate_immediately).to be_falsey |
| 314 | 365 |
expect(weather_agent).to be_disabled |
| 315 | 366 |
expect(weather_agent.memory).to be_empty |
@@ -330,7 +381,7 @@ describe ScenarioImport do |
||
| 330 | 381 |
"0" => {
|
| 331 | 382 |
"name" => "updated name", |
| 332 | 383 |
"schedule" => "6pm", |
| 333 |
- "keep_events_for" => "2", |
|
| 384 |
+ "keep_events_for" => 2.days.to_i.to_s, |
|
| 334 | 385 |
"disabled" => "false", |
| 335 | 386 |
"options" => weather_agent_options.merge("api_key" => "foo").to_json
|
| 336 | 387 |
} |
@@ -343,7 +394,7 @@ describe ScenarioImport do |
||
| 343 | 394 |
weather_agent = existing_scenario.agents.find_by(:guid => "a-weather-agent") |
| 344 | 395 |
expect(weather_agent.name).to eq("updated name")
|
| 345 | 396 |
expect(weather_agent.schedule).to eq("6pm")
|
| 346 |
- expect(weather_agent.keep_events_for).to eq(2) |
|
| 397 |
+ expect(weather_agent.keep_events_for).to eq(2.days.to_i) |
|
| 347 | 398 |
expect(weather_agent).not_to be_disabled |
| 348 | 399 |
expect(weather_agent.options).to eq(weather_agent_options.merge("api_key" => "foo"))
|
| 349 | 400 |
end |
@@ -353,7 +404,7 @@ describe ScenarioImport do |
||
| 353 | 404 |
"0" => {
|
| 354 | 405 |
"name" => "", |
| 355 | 406 |
"schedule" => "foo", |
| 356 |
- "keep_events_for" => "2", |
|
| 407 |
+ "keep_events_for" => 2.days.to_i.to_s, |
|
| 357 | 408 |
"options" => weather_agent_options.merge("api_key" => "").to_json
|
| 358 | 409 |
} |
| 359 | 410 |
} |
@@ -386,12 +437,40 @@ describe ScenarioImport do |
||
| 386 | 437 |
end |
| 387 | 438 |
end |
| 388 | 439 |
|
| 440 |
+ context "when the schema_version is less than 1" do |
|
| 441 |
+ it "translates keep_events_for from days to seconds" do |
|
| 442 |
+ valid_parsed_data.delete(:schema_version) |
|
| 443 |
+ valid_parsed_data[:agents] = [valid_parsed_weather_agent_data.merge(keep_events_for: 5)] |
|
| 444 |
+ |
|
| 445 |
+ scenario_import.merges = {
|
|
| 446 |
+ "0" => {
|
|
| 447 |
+ "name" => "a new name", |
|
| 448 |
+ "schedule" => "6pm", |
|
| 449 |
+ "keep_events_for" => 2.days.to_i.to_s, |
|
| 450 |
+ "disabled" => "true", |
|
| 451 |
+ "options" => weather_agent_options.merge("api_key" => "foo").to_json
|
|
| 452 |
+ } |
|
| 453 |
+ } |
|
| 454 |
+ |
|
| 455 |
+ expect(scenario_import).to be_valid |
|
| 456 |
+ |
|
| 457 |
+ weather_agent_diff = scenario_import.agent_diffs[0] |
|
| 458 |
+ |
|
| 459 |
+ expect(weather_agent_diff.name.current).to eq(agents(:bob_weather_agent).name) |
|
| 460 |
+ expect(weather_agent_diff.name.incoming).to eq('a weather agent')
|
|
| 461 |
+ expect(weather_agent_diff.name.updated).to eq('a new name')
|
|
| 462 |
+ expect(weather_agent_diff.keep_events_for.current).to eq(45.days.to_i) |
|
| 463 |
+ expect(weather_agent_diff.keep_events_for.incoming).to eq(5.days.to_i) |
|
| 464 |
+ expect(weather_agent_diff.keep_events_for.updated).to eq(2.days.to_i.to_s) |
|
| 465 |
+ end |
|
| 466 |
+ end |
|
| 467 |
+ |
|
| 389 | 468 |
it "sets the 'updated' FieldDiff values based on any feedback from the user" do |
| 390 | 469 |
scenario_import.merges = {
|
| 391 | 470 |
"0" => {
|
| 392 | 471 |
"name" => "a new name", |
| 393 | 472 |
"schedule" => "6pm", |
| 394 |
- "keep_events_for" => "2", |
|
| 473 |
+ "keep_events_for" => 2.days.to_s, |
|
| 395 | 474 |
"disabled" => "true", |
| 396 | 475 |
"options" => weather_agent_options.merge("api_key" => "foo").to_json
|
| 397 | 476 |
}, |
@@ -411,7 +490,8 @@ describe ScenarioImport do |
||
| 411 | 490 |
expect(weather_agent_diff.name.updated).to eq("a new name")
|
| 412 | 491 |
|
| 413 | 492 |
expect(weather_agent_diff.schedule.updated).to eq("6pm")
|
| 414 |
- expect(weather_agent_diff.keep_events_for.updated).to eq("2")
|
|
| 493 |
+ expect(weather_agent_diff.keep_events_for.current).to eq(45.days) |
|
| 494 |
+ expect(weather_agent_diff.keep_events_for.updated).to eq(2.days.to_s) |
|
| 415 | 495 |
expect(weather_agent_diff.disabled.updated).to eq("true")
|
| 416 | 496 |
expect(weather_agent_diff.options.updated).to eq(weather_agent_options.merge("api_key" => "foo"))
|
| 417 | 497 |
end |
+ 25
- 7
spec/models/users_spec.rb
@@ -3,15 +3,33 @@ require 'spec_helper' |
||
| 3 | 3 |
describe User do |
| 4 | 4 |
describe "validations" do |
| 5 | 5 |
describe "invitation_code" do |
| 6 |
- it "only accepts valid invitation codes" do |
|
| 7 |
- User::INVITATION_CODES.each do |v| |
|
| 8 |
- is_expected.to allow_value(v).for(:invitation_code) |
|
| 6 |
+ context "when configured to use invitation codes" do |
|
| 7 |
+ before do |
|
| 8 |
+ stub(User).using_invitation_code? {true}
|
|
| 9 |
+ end |
|
| 10 |
+ |
|
| 11 |
+ it "only accepts valid invitation codes" do |
|
| 12 |
+ User::INVITATION_CODES.each do |v| |
|
| 13 |
+ is_expected.to allow_value(v).for(:invitation_code) |
|
| 14 |
+ end |
|
| 15 |
+ end |
|
| 16 |
+ |
|
| 17 |
+ it "can reject invalid invitation codes" do |
|
| 18 |
+ %w['foo', 'bar'].each do |v| |
|
| 19 |
+ is_expected.not_to allow_value(v).for(:invitation_code) |
|
| 20 |
+ end |
|
| 9 | 21 |
end |
| 10 | 22 |
end |
| 11 |
- |
|
| 12 |
- it "can reject invalid invitation codes" do |
|
| 13 |
- %w['foo', 'bar'].each do |v| |
|
| 14 |
- is_expected.not_to allow_value(v).for(:invitation_code) |
|
| 23 |
+ |
|
| 24 |
+ context "when configured not to use invitation codes" do |
|
| 25 |
+ before do |
|
| 26 |
+ stub(User).using_invitation_code? {false}
|
|
| 27 |
+ end |
|
| 28 |
+ |
|
| 29 |
+ it "skips this validation" do |
|
| 30 |
+ %w['foo', 'bar', nil, ''].each do |v| |
|
| 31 |
+ is_expected.to allow_value(v).for(:invitation_code) |
|
| 32 |
+ end |
|
| 15 | 33 |
end |
| 16 | 34 |
end |
| 17 | 35 |
end |
+ 33
- 0
spec/support/shared_examples/web_request_concern.rb
@@ -141,5 +141,38 @@ shared_examples_for WebRequestConcern do |
||
| 141 | 141 |
agent.options['disable_ssl_verification'] = false |
| 142 | 142 |
expect(agent.faraday.ssl.verify).to eq(true) |
| 143 | 143 |
end |
| 144 |
+ |
|
| 145 |
+ it "should use faradays default params_encoder" do |
|
| 146 |
+ expect(agent.faraday.options.params_encoder).to eq(nil) |
|
| 147 |
+ agent.options['disable_url_encoding'] = 'false' |
|
| 148 |
+ expect(agent.faraday.options.params_encoder).to eq(nil) |
|
| 149 |
+ agent.options['disable_url_encoding'] = false |
|
| 150 |
+ expect(agent.faraday.options.params_encoder).to eq(nil) |
|
| 151 |
+ end |
|
| 152 |
+ |
|
| 153 |
+ it "should use WebRequestConcern::DoNotEncoder when disable_url_encoding is truthy" do |
|
| 154 |
+ agent.options['disable_url_encoding'] = true |
|
| 155 |
+ expect(agent.faraday.options.params_encoder).to eq(WebRequestConcern::DoNotEncoder) |
|
| 156 |
+ agent.options['disable_url_encoding'] = 'true' |
|
| 157 |
+ expect(agent.faraday.options.params_encoder).to eq(WebRequestConcern::DoNotEncoder) |
|
| 158 |
+ end |
|
| 159 |
+ end |
|
| 160 |
+ |
|
| 161 |
+ describe WebRequestConcern::DoNotEncoder do |
|
| 162 |
+ it "should not encode special characters" do |
|
| 163 |
+ expect(WebRequestConcern::DoNotEncoder.encode('GetRss?CategoryNr=39207' => 'test')).to eq('GetRss?CategoryNr=39207=test')
|
|
| 164 |
+ end |
|
| 165 |
+ |
|
| 166 |
+ it "should work without a value present" do |
|
| 167 |
+ expect(WebRequestConcern::DoNotEncoder.encode('GetRss?CategoryNr=39207' => nil)).to eq('GetRss?CategoryNr=39207')
|
|
| 168 |
+ end |
|
| 169 |
+ |
|
| 170 |
+ it "should work without an empty value" do |
|
| 171 |
+ expect(WebRequestConcern::DoNotEncoder.encode('GetRss?CategoryNr=39207' => '')).to eq('GetRss?CategoryNr=39207=')
|
|
| 172 |
+ end |
|
| 173 |
+ |
|
| 174 |
+ it "should return the value when decoding" do |
|
| 175 |
+ expect(WebRequestConcern::DoNotEncoder.decode('val')).to eq(['val'])
|
|
| 176 |
+ end |
|
| 144 | 177 |
end |
| 145 | 178 |
end |
+ 1
- 1
spec/support/vcr_support.rb
@@ -2,7 +2,7 @@ require 'vcr' |
||
| 2 | 2 |
|
| 3 | 3 |
VCR.configure do |c| |
| 4 | 4 |
c.cassette_library_dir = 'spec/cassettes' |
| 5 |
- c.allow_http_connections_when_no_cassette = true |
|
| 5 |
+ c.allow_http_connections_when_no_cassette = false |
|
| 6 | 6 |
c.hook_into :webmock |
| 7 | 7 |
c.default_cassette_options = { record: :new_episodes}
|
| 8 | 8 |
c.configure_rspec_metadata! |