High Level Roadmap
- GRIBNASTY - Pull directly from GRIBs / Ditch APIs
- Map Improvements - show more airport info, fly vs. drive, etc.
- Misc - UI fixes, colors, optimization
Figure out the db schematic to allow multiple data sources for each parameter.
- Is it possible to store each forecast field/value in a single table and then do a mega query on it to pull in exactly what we want as well as do avg/min/max/stddev.
- GFS = 192 hours out / 3hr interval = 64 points. Although if we interpolate, then back to 192
- 2000 spots + 4000 airports + room for another 1000 spots = 7000 'locations'
- 2 or 3 sources for each parameter (gfs vs ecmwf vs nam, ww3 vs wam)
- ~10 parameters we find useful
- 7000 192 3 * 10 = 40,320,000 entries. maaaaybe? gonna need to be pretty damn fucking optimized
- Might be able to split it into separate tables for each parameter (gets us donwn to ~4M entries per parameter)
- big advantage of per-hour parameters is we can custom filter per user on a per-hour basis and calculate kiteable hours, avg/day, etc.
- If not per-hour entries, then per-parameter will definitely work
- 7000 192 3 = 4,032,000 entries very doable
re-investigate scoring of conditions. some variables:
- kiteable == >=13kts?
- Is it possible to make this a completely user configurable parameter based on filters / search?
- kiteable hours per day (daily)
- average kiteable hours/day (forecast and historical)
- kiteable == >=13kts?
Add data sources page to user profile
- Allow user to select which data source for each parameter (wind_speed, air_temperature, sea_temperature, etc)
- Do we want to make this per-parameter, or more logical (eg wind, swell, etc.)
3835 airports (1 call/day) ~2000 spots * (2 call/day)
3835 + 4000 = 7835 calls/day
- Darksky - per-query (unlimited)
- WorldWeatherOnline - 15k/day
- PlanetOS - 3.3k/day (???)
- UGRD -> :UGRD:10 m above ground:
- VGRD -> :VGRD:10 m above ground:
- GUST -> :GUST:surface:
- TMP -> :TMP:surface:
- PRATE -> :PRATE:surface:
- TCDC -> :TCDC:entire atmosphere:0-1 hour ave fcst:
//new GFS released: UTC 00, 06, 12, 18 $hour = floor(date("G") / 6) * 6;
$next = date("Ymd
(WIND / WDIR)
UGRD -> :UGRD:10 m above ground:
VGRD -> :VGRD:10 m above ground:
GUST -> :GUST:surface:
TMP -> :TMP:surface:
PRATE -> :PRATE:surface:
TCDC -> :TCDC:entire atmosphere:0-1 hour ave fcst:
- Try phptimezone.php class (https://github.com/webmasterar/TimezoneMapperPHP)
- Make a script to lookup each spots timezone and compare with current
- Find where we're accessing timezone stuff and make sure it all flows from spot timezone.
Astronomy (sunrise/sunset, moonrise/moonset, moon phase)
- php has date_sunrise and date_sunset functions
- this class seems full featured: https://github.com/gregseth/suncalc-php
- this looks legit: https://rapidapi.com/apihood/api/tides/details
- potential backup: https://github.com/joelkoz/xtwsd
API: Google Timezone
- Used: timezone
API: Darksky (used for lots)
Parsed: sunrise/sunset, moon phase, wind, air temp, sea temp, cloud cover, apparent temp, dewpoint, humidity, pressure, UV index, ozone, precipitation Used: moon phase, wind, air temp, clouds, precip, Backup For: sunrise/sunset, sea temp, sunrise/sunset, swell
Parsed: sunrise/sunset, moonrise/moonset, tides, wind, air temp, precip, cloud cover, swell, sea temp Used: tides, sea temp, sun/moon times Backup For: swell Unused: wind, air temp, precip, clouds
API: PlanetOS (used for WaveWatch 3 data)
Parsed: wind speed, swell Used: swell Backup For: none Unused: wind speed
Long Term Wants
More Kite Spots
Prerequisites - move away from APIs
- moar australia - south coast?
- moar madagascar
- moar US east coast
- Woo daily leaderboards
- surf spots from magicseaweed
- surf spots from surfline
- load airport links on per-airport basis, load them when clicking on airport icon.
- unwind airport stuff from spot search megasql query - use airport links in search.
- Add 'my location' map selector to settings page
- Show "my location" on travel map
- integrate windy api as a layer
- Show travel time when highlighting route line
- figure out how to integrate location + airport + flyable + driveable into single travel view.
- figure out how to show wave stats - yin/yang idea?
- Draw hops on /map page (location to spot, location to airport, airport to airport, airport to spot)
- Add highlighting functionality to map (when highlight over link to airport, spot then highlight relevant marker)
- remove airport_id and airport_code from spot
- international date line handling: -180/180 (spot: 303) (in what part of site?)
- add ignored spots
- add ignored countries / regions
- add spot ratings
- add spot comments
True Time Calculation
- added country_id to spots - flow to rest of site
- refactor getKiteable to handle lookup and merging of spot stuff, not controller (maybe create a newer, higher level function)
- add lookup of all major airports within 50km of current location for map page, not just one (on the session level...)
- as part of lookup, find out travel time from current location to each airport (via google matrix api)
- add list of spot links to airport page
UI / Site Upgrades +++
- add preferred/kiteable wind directions to spots
- add to spot record: type - is_flat / is_waves / is_foilable
- have a 1-5 star rating?
- add in airline detail page
- good UI: http://www.boracayweatherstation.com/
- Experiment with light theme to make bootstrap work better.
- Compass Rose / Cardinal Diretion selector:
Framework / Optimization
- start compressing all forecast data with gzcompress -9
- optimize mysql configuration
- optimize apache configuration
- optimize memcached configuration
- add optional memcache support for to collections object
- add memcache support to airport / spot search collections
- bust airport/spot searches when new forecasts are loaded, as well as when a spot is added/edited.
- add proper page validation handling via PageException
- update form class to conform to bootstrap 4 style - add elements by row / field / label
- is it worth removing filters completely and just having that functionality baked into searches only?
- probably not, its nice to have filters that can be reused across different searches
- might be interesting long term to have the filter data simply pulled in and then the modal shows the sliders that can then be changed.
- user can then just either update the search, save the filter, or create a new filter.
- gotta figure out how to not make that confusing. should probably be its own independent update.
Cool / Potential APIs
https://api.video https://openweathermap.org/current https://www.ncdc.noaa.gov/cdo-web/webservices/v2#datasets https://github.com/planetopendata/awesome-world Air Quality: http://aqicn.org/json-api/doc/ (key: d921c0adae82a6b2f48800b24d02d0dd53767a8c) GDP Per Capita on Country: https://data.worldbank.org/indicator/NY.GDP.PCAP.CD
- Potential: https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt
- Potential: https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/gsom-gsoy_documentation.pdf
- Potential: https://www.ncei.noaa.gov/data/marine/icoads3.0/enhanced-trim/ICOADS_r3.0.0_README_EnhancedTrimming_c20170810.pdf
Documentation / GRIB
- grib format / wgrib2 info - ftp://ftp.cpc.ncep.noaa.gov/wd51we/ams2010_sc/formats.pdf
- grib2 codes - https://ruc.noaa.gov/ruc/ruc20gribtable.html
- grib2 products - http://www.nco.ncep.noaa.gov/pmb/products/cfs/
- wgrib2 - http://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/index.html
- WaveWatch III Hindcast: ftp://polar.ncep.noaa.gov/pub/history/waves/multi_1/00README
- Sea Surface Temps Historical: ftp://polar.ncep.noaa.gov/history/sst/climatology_1981_2011/README
- CFS Re-analysis - https://www.ncdc.noaa.gov/data-access/model-data/model-datasets/climate-forecast-system-version2-cfsv2
- GFS realtime forecast site - http://nomads.ncep.noaa.gov/
- GFS 0.25deg hourly forecast - http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25_1hr.pl?dir=%2Fgfs.2018091206
- Raw GFS - http://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod
- Color Theory: https://www.climate-lab-book.ac.uk/2014/which-colour-scale/
- created a sitemap.xml file and updated robots.txt
- pushed search/spot alerts to production
- made alert emails a bit nicer looking
- cleaned up and added confirmations to various delete pages
- fixed share search icon
- wrote the spot alert cronjob script
- warn on filter delete if it has alerts / searches / users
- started work on spot alert system
- search alert cronjob is mostly done
- made email a bit nicer looking
- fleshed out the search alert CRUD system
- started work on the search alerts feature
- a few minor tweaks to search system before launch
- user interface is all working nicely together now
- added user admin area to manage/delete searches
- made big progress on building the search sharing functionality
- added links on airport page to different info sites.
- changed spot favorite UI to inline loading bubble ajax
- added a link to meteoearth
- finally cleaned up and moved historical scripts into cronjob
- added link to iGetwind on spot view page
- added favorite spot functionality
- integrated min/max/stddev value params to search filters
- make public Roadmap / Changelog page
- show searched column on search results
- allow sort order changing on column click
- use default sort order on column click if changing order param
- add std dev to historical stats page
- move forecast stats to their own tab
- move more parameter displays to using the new main::forecast_field view
- removed old monthlybest and airport search pages
- made individual color gradients and ranges for each std dev parameter nicer
- Fixed forecast db entry creation so we dont have multiple old forecasts hanging around.
- Added standard deviation to all averaged params.
- Started on structure for selecting sort order
- integrate new min/max stuff into historical data
- fixed a small bug with the travel time search order
- FIXED: min/max are not on an hour-by-hour basis
- changed all camelCase weather params to under_score for consistency
- integrate wave power into historical data
- verify historical data params are all camel_case
- show swell power when sorting by swell power
- show swell power + swell period on historical pages
- added swell power field to forecasts
- add sorting on swell power and certain max parameters
- get min/max forecast db values updating properly
- show min/avg/mag on forecast page
- added links to other weather services
- figure out how to make historical-3-crunch-spot-data more efficient so it can be run as a cronjob.
- add verified/unverified icon to spot maker infobox + load spot forecast by ajax
- add verified/unverified filter to search + default user search params
- hoekens filter - public
- lat/lon picker is fixed
- region/country/airport/spot map pages work nicely now
- fixed bug where spot moderation emails get sent every edit
- show hidden spots on detailed list like on hybrid
- sort icons now work on both headers
- show hidden spots on map as low opacity + info in box
- re-enabled travel search and all other search params - hidden is now handled by controller
- when changing filter/group/etc reset page to 1
- standardize loading gif between map and search
- fix "your airport" box to have better spacing on all pages
- updated TODO a bit
- changed precip coloring on hourly forecast
- started framework for displaying 'hidden' spots
- add ability to suggest a spot for non-admin users
- add spot approve page
- add spot delete page
- send email to admins when new spot suggested
- update hybrid view to reload table when scrolling
- show a greyed out loading screen when updating spots map on move.
- when adding new spot and no airport links found, look for closest airport
- add links based on distance when no google distance found, show to the user too.
- now will pull airports from the spot itself, added a flag to show airports or not.
- make map reload work with filters / search params
- load new spots on map move
- refactored KiteMap to WindNinjaMap to make it easier and more standardized
- upgraded all instances of KiteMap to WindNinjaMap
- update spotpicker to use new WindNinjaMap js object
- add ability to right-click on map to add a new spot
- automatially set airport + country based on location
- add verified boolean checkbox to spots
- add verified location icon
- remove commute time from spot
- make sure GeoIP stuff is working properly - yep, checks out!
- combine spot create and spot edit controller logic
- consolidated global files
- attempted to fix the geoip lookup stuff
- change forecast to 7 days instead of 5
- fix airport pagination
- added a stacked bar graph that shows BFT % by month (and waves too)
- add region > country > airport to spots page
- add region to country page
- add region > country to airport page
- add a "your location: Region > Country > Airport" on template header
- added historical links to navbar and forecast page
- fixed prate importing (hopefully)
- a few small tweaks for mobile
- have done a ton of work on getting historical data imported. almost done!
- revamped the spot grouping setup to make searches more accurate
- spot search now works with historical data!!!
- a couple little tweaks here and there to clean up post-launch
- added links to settings for search controls.
- updated user login/registration to be a bit nicer.
- clean up snippets of js in random spots.
- Make sure new site works okay in mobile.
- upgrade server software
- added loading graphic on spot load ajax
- hide spot display panel and add link in header to show it
- update filter to have checkbox to enable each parameter, and use that when building search query
- fixed country and airport pages back to how it was before.
- travel map on airport page should default to that airport.
- when travel filter is on, show travel time on hybrid/list
- added travel filter defaults to location setting page.
- worked a ton on the new AJAX UI - its working well.
- changed spot view type to dropdown
- move sort/group stuff from filters to user
- default group by on country and airport view should be none
- worked on filter controls UI
- fixed airport / country display maps
- Make airport incoming/outgoing flights a tabbed view
- Add pagination to airport list on country page
- Add pagination to airport incoming/outgoing flights list
- add a 'sort/cluster' filter bar (cluster by country, airport)
- Bonaire Saint Eustatius And Saba - whats up with that? Should probably be "Netherlands Antilles"
- Fixed Sint Maarten / Saint Marten airport bug
- Create a few good defaults like 'No Wetsuit Needed' or 'Strong Wind Only'
- allow sorting by wind speed / wave height / air temp / sea temp by clicking on column headers
- added tooltips to emojis for clarity
- added tooltips to dates on forecast for clarity on day of week.
- allow user to set 'no filter' and site defaults as default filter
- allow user to set default page type on profile level, not filter
- reworked filter controls to allow page change as well as filter selection.
- remove 'site default filter'
- default to no filter
- travel map can now display spots in list, map, hybrid forms
- refactor to use a common codebase for pulling in filter id, page, for the various pages that all use filters.
- pagination works on all filter pages w/ lists.
- we should default to no filter, and have a couple good defaults
- Settings is now a dropdown to each individual settings tab
- consolidate all filterable spot list views into the spots controller
- added regions functionality: https://www.internetworldstats.com/list1.htm
- moved appropriate countriest to the appropriate region
- removed continent links
- added filters to airports view page
- Add pagination to filter view pages: list / hybrid
- Update settings url linkage to allow linking straight to a tab
- Working on filters UI element
- Added filters + pagination to country, windiest, and waviest pages
- Also got travel map working with filters.
- Added filter info modal dialog to filter view page
- Make travel map view actually work...
- Move away from kitemap API call to travelmap / spotmap API calls.
- Created filter list/map/hybrid/travel view pages
- worked on filter edit page
- added filter view page
- Add CRUD functionality to manage filters
- add settings page - what homepage, email, airport, location
- tweaked map page to pull from user profile data
- clean up the nav a bit.
- add admin / logged in asserts to proper pages.
- added spot / airport duration into formula when calculating spots based on distance
- add google matrix lookup of all spots within (hours * 100 km) of current location / airport
- transition to airport_id instead of iata / airport_code
- add distance/time lookup between spot and airport
- create reserved server on amazon ec2
- create script to truncate huge forecasts table
- create main site config.php
- create main site virtualhost entry
- created define to allow the site to be taken down
- transitioned site to new ec2 server
- fixes here and there for moving to the amazon ec2 server
- make table of monthly historical links / data for spot history page
- make historical forecast month detail page w/ links to individual historical forecasts
- add a historical page which will show the historical data for each spot / airport
- add historical summary page which shows the best spots per month
- added a 'wave rose' in addition to the wind rose.
- fix wind/wave direction averaging: https://math.stackexchange.com/questions/44621/calculate-average-wind-direction
- pull in tide information from WWO forecast
- pull in moonrise/moonset information from WWO forecast
- display sunrise/sunset on forecast
- display moonrise/moonset/moon phase on forecast
- display tide information on forecast
- use emojis to display moon phase: 🌑 🌒 🌓 🌔 🌕 🌖 🌗 🌘 🌑
- pull in all the rest of the variables from all forecast providers (pressure, etc)
- add z-height on map spot markers based on wind speed
- round the corners of map spot markers
- fixed a bug with timezone parsing
- double check sunrise/sunset stuff on TOS airport - yep, no sunrise/sunset!
- fixed kite map display bug when zero spots and tweaked UI a bit.
- fixed a bug on daily display when local time is near midnight
- change hourly forecast to pull in all hours in the controller, check to make sure none are before now()
- fixed some minor bugs with forecast display
- Move to Forecast -> RawForecast system
- Forecast is a single object with every potential data, parsed into a common format
- Raw Forecast is a specific forecast from an individual forecast provider (eg. DarkSky, WWO, PlanetOS, etc)
- Then Forecast will loop through each one and generate the metaforecast
- forecast detail page now shows an hourly forecast and is actually useful now
- tidied up scripts directory
- fixed some bugs in forecasts, now pulling in hourly wave forecasts too
- refactored map js to be a bit more universal
- add line between spot and airport on map view
- add wave spots mode to kite map
- show top spots on kite map
- added in initial support for planetOS WW3 data
- figure out timezone based on lat/lon coordinates: https://developers.google.com/maps/documentation/timezone/start
- changed to group by country on windiest/waviest
- is_unkiteable airports should have a greyed out icon
- add nearby airports (and spots) to airport view page
- a couple more tweaks to route view page
- revamped route page to be more concise + better links + more info
- optimize spot map ajax loading to not recreate teh entire map, lol
- show top spots for each continent on continent view page
- add map view to continent view page
- db backups now happening daily!
- mobile views of spots now with emojis and better lookign color. :)
- on kite map page, an enter key press on airport should trigger teh ajax call
- show description text on spot detail page (now with Markdown support!)
- add geoip support to guess where the user is from
- guess closest airport to the user based on lat/lon
- add in a link to airport page from map on country detail page
- moved all maps to use new map code
- fix zoom on country map page
- fix quote escape bug on map pages
- move main map controls to map UI
- convert main map page to be AJAX/JSON and dynamic
- crushed out the first iteration of the mega map!
- get megaerror functionality working
- add continent to search parameters
- add continent view page - list of countries
- add link to country from airport page
- lots of little site tweaks/updates
- spot search now using parameterized search.
- imported a real country database from http://geonames.org
- add airport listing to country view page
- add unkiteable field to airport
- fixed kite forecast loading on add/edit due to whitespace in lat/lon fields
- update route derivation to not add intermediate flights and only add nonstop flights.
- added kiteable page for airports
- changed homepage to top 25 spots worldwide
- add lat/lon min/max for map lookups
- support multiple source airports in search
- fixed premade airport searches
- tweaked airport map page to not be super zoomed
- removed dead data from forecast display
- added route detail page w/ info on segments
- added ordering on flight distance, added display of flight distance.
- add ability to search airports by # of flights, altitude, and country
- add ability to search airports by duration/hops from an airport
- add incoming/outgoing flight count to airport search form
- added favicon
- added spots from: https://www.tripping.com/explore/the-10-best-places-for-kiteboarding-in-the-world
- added precipitation and cloud cover color formatting
- add field to airports to hold incoming/outgoing flight count
- verify precipitation field units/total
- better colors on winds + match colors from windguru
- fixed cronjob script to clean up forecasts
- update score field globally when pulling in spot forecasts
- change all spot pages to use ranking on search
- add in custom airport query pages: rainiest, snowiest, hottest, coldest, kiteable, nice weather, etc.
- add cloud cover to global airport search
- add google analytics code for tracking
- re-add footer to main template
- add about page that explains whats up