todo.txt

Server / API Middleware:
    - When it can, the middleware should automatically convert incoming JSON requests to their javascript model objects,
      for convenience
    - (FUTURE) self documenting endpoints
    - Going to /app should redirect you to /app/
    - None of the API endpoints should contain dashes in them. better to be pure alphanumeric for maximum compatibility with other systems



Schemas:
    - Need to implement full support of vanilla arrays, e.g. arrays that are just strings or numbers instead of objects
    - Implement a map() and mapObject() functions, both with async versions
    - Implement union() and difference() functions
    - Make sure that every walk/climb/transform function has both synchronous and asynchronous versions, where applicable
    - A convenient function should be created to map metadata from one schema to another similar schema. This can be used
      so that configuration doesn't get lost, even if a schema changes somewhat in an unpredictable fashion
    - Convenience methods to lookup up a particular schema within the tree when given a variablePath
    - Convenience methods to validate objects and to filter out anything not in the schema
    - Create a function that returns a flattened version of the schema. This is just where all unnecessary sub-objects are flattened
      so that their fields are part of the parent object. This should also have convenient way of converting real objects from their
      full version to their flattened version
    - Need to make all of the walk-schema methods recurse simply by calling their own walk-schema method, instead
      of recursing internally.
    - Preferably, walk object has some graceful error handling if the object does not conform to the schema
    - (MAYBE) Every schema object should keep track of both the original schema, and the new schema that results after
      transformations have been applied.
    - Automatic schema detection should be a background task. The admin should present a progress bar with some
      diagnostic information.
    - Should have thorough unit tests for all the schema manipulation functions
    - Read the documentation for underscore.js, and try and come up with more useful utility functions that we could code
      that would apply to schemas
    - (Maybe) Implement native support for binary data
    - We should have a set of convenience functions for validating that a schema conforms to some set of requirements. For example,
      checking if it only contains numbers or arrays, a requirement for inputs to neural networks. Or checking if it is flat - having
      no extra object sub-schemas that aren't necessary.
    - We should separate configuration for a particular schema entry, like its transformation which is configured by the user,
      from the metadata for that schema entry, such as cardinality and known values which are determined automatically by the
      schema detection code.
    - It would be nice if Schemas became a first-class object within electric brain. This way, when creating an architecture, you
      don't choose a data source. You choose the schema from that data source. Later when training a model on that architecture,
      you have the option of selecting different data sources as long as they produce the same schema
    - Merge the function in schema_utilities.js into the EBSchema class
    - We need to make a hard linguistic distinction between "schema" and "field". Currently we use them interchangeably, but
      moving forward, we should only use "field" when the schema question is for a single field, like a string or number, vs.
      an array or object.
    - Schema differencing function? (might be useful someday)



New Features:
    Data Source:
        - Be able to enter an arbitrary mongo query, so that we can filter which objects are included in the data source.
        - Able to populate referenced documents
        - Should be able to have a "whole object" transformation that processes all objects that come out of the data source
    Architecture -> Configure Inputs:
        - Implement the preview button. Find a new place for it on the interface. Maybe "preview" functionality
          should be general purpose, so that it could be applied to the data-source screen and the configure
          output screen as well
        - Would be nice if we could preview just one fields transformation, rather then the object as a whole
        - The system should verify that all the outputs conform to the required format for neural networks, e.g.
          that every field is either a number, object or an array. It should present an error to the user if this
          is not the case, along with diagnostic information.
    Front End:
        - We need a general purpose error handler for all unsuccessful HTTP calls