Merge pull request #2 from ussserrr/dev

v0.9

CHANGELOG

@@ -86,6 +86,7 @@ stm32pio changelog:
         - New: add PyCharm to .gitignore
         - New: add clear TODOs for the next release (some sort of a roadmap)
         - New: single __version__ reference
+        - New: extended shebang
         - New: add some new tests (test_build_should_raise, test_file_not_found)
         - Fixed: options '--start-editor' and '--with-build' can now be used both for 'new' and 'generate' commands
         - Fixed: import scheme is now as it should be
@@ -96,6 +97,31 @@ stm32pio changelog:
         - Changed: logging output in standard (non-verbose) mode is simpler
         - Changed: move tests in new location
         - Changed: revised and improved tests
-        - Changed: tests are now run by 'python -m unittest' and cannot be run as standalone
         - Changed: actualized .ioc file and clean-up the code according to the latest STM32CubeMX version (5.3.0 at the moment)
         - Changed: revised and improved util module
+
+    ver. 0.9 (12.19):
+        - New: tested with Python3 version of PlatformIO
+        - New: '__main__.py' file (to run the app as module (python -m stm32pio))
+        - New: 'init' subcommand (initialize the project only, useful for the preliminary tweaking)
+        - New: introducing the OOP pattern: we have now a Stm32pio class representing a single project (project path as a main identifier)
+        - New: projects now have a config file stm32pio.ini where the user can set the variety of parameters
+        - New: 'state' property calculating the estimated project state on every request to itself (beta). It is the concept for future releases
+        - New: STM32CubeMX is now started more silently (without a splash screen)
+        - New: add integration and CLI tests (sort of)
+        - New: testing with different Python versions using pyenv (3.6+ target)
+        - New: 'run_editor' test is now preliminary automatically check whether an editor is installed on the machine
+        - New: more typing annotations
+        - Fixed: the app has been failed to start as 'python app.py' (modify sys.path to fix)
+        - Changed: 'main' function is now fully modular: can be run from anywhere with given CLI arguments (will be piped forward to be parsed via 'argparse')
+        - Changed: rename stm32pio.py -> app.py (stm32pio is the name of the package as a whole)
+        - Changed: rename util.py -> lib.py (means main library)
+        - Changed: logging is now more modular: we do not set global 'basicConfig' and specify separated loggers for each module instead
+        - Changed: more clear description of steps to do for each user subcommand by the code
+        - Changed: get rid of 'print' calls leaving only logging messages (easy to turn on/off the console output in the outer code)
+        - Changed: re-imagined API behavior: where to raise exceptions, where to return values and so on
+        - Changed: more clean API, e.g. move out the board resolving procedure from the 'pio_init' method and so on
+        - Changed: test fixture is now moved out from the repo and is deployed temporarily on every test run
+        - Changed: set-up and tear-down stages are now done using 'unittest' API
+        - Changed: actualized .ioc file for the latest STM32CubeMX version (5.4.0 at the moment)
+        - Changed: improved help, docs, comments

MANIFEST.in

@@ -1,8 +1,8 @@
+include MANIFEST.in
 include README.md
 include LICENSE
-include MANIFEST.in
 include CHANGELOG
 include TODO.md
 include .gitignore
-recursive-include stm32pio/tests *
+recursive-include stm32pio-test-project *
 include screenshots/*.png

README.md

@@ -1,63 +1,64 @@
 # stm32pio
 Small cross-platform Python app that can create and update [PlatformIO](https://platformio.org) projects from [STM32CubeMX](https://www.st.com/en/development-tools/stm32cubemx.html) `.ioc` files.
 
+It uses STM32CubeMX to generate a HAL-framework based code and alongside creates PlatformIO project with the compatible `stm32cube` framework specified.
+
 ![Logo](/screenshots/logo.png)
 
 
 ## Features
   - Start the new project in a single directory using only an `.ioc` file
   - Update existing project after changing hardware options from CubeMX
-  - Clean-up the project (WARNING: it deletes ALL content of 'path' except the `.ioc` file!)
+  - Clean-up the project (WARNING: it deletes ALL content of project path except the `.ioc` file!)
   - *[optional]* Automatically run your favorite editor in the end
-  - *[optional]* Make an initial build of the project
-
-
-## Restrictions
-  - The tool doesn't check for different parameters compatibility, e.g. CPU frequency, memory sizes and so on. It simply eases your workflow with these 2 programs (PlatformIO and STM32CubeMX) a little bit.
-  - CubeMX middlewares don't support yet because it's hard to be prepared for every possible configuration. You need to manually adjust them to build appropriately. For example, FreeRTOS can be added via PlatformIO' `lib` feature or be directly compiled in its own directory using `lib_extra_dirs` option:
-    ```ini
-    lib_extra_dirs = Middlewares/Third_Party/FreeRTOS
-    ```
-    You also need to move all `.c`/`.h` files to the `src`/`include` folders respectively. See PlatformIO documentation for more information.
+  - *[optional]* Automatically make an initial build of the project
 
 
 ## Requirements:
   - For this app:
-    - Python 3.6+
+    - Python 3.6 and above
   - For usage:
     - macOS, Linux, Windows
-    - STM32CubeMX (all recent versions) with downloaded necessary frameworks (F0, F1, etc.). Try to generate code in ordinary way (through the GUI) at least once before running stm32pio
-    - Java CLI (JRE) (likely is already installed if STM32CubeMX works)
-    - PlatformIO CLI.
+    - STM32CubeMX (all recent versions) with desired downloaded frameworks (F0, F1, etc.)
+    - Java CLI (JRE) (likely is already installed if the STM32CubeMX is working)
+    - PlatformIO CLI (already presented if you have installed PlatformIO via some package manager or need to be installed as the command line extension from IDE)
+
+A general recommendation there would be to try to generate and build a code manually (via the CubeMX GUI and PlatformIO CLI or IDE) at least once before using stm32pio to make sure that all tools are working properly without any "glue".
+
+
+## Installation
+Starting from v0.8 it is possible to install the utility to be able to run stm32pio from anywhere. Use
+```shell script
+stm32pio-repo/ $   pip3 install .
+```
+command to launch the setup process. To uninstall run
+```shell script
+$ pip3 uninstall stm32pio
+```
 
 
 ## Usage
 Basically, you need to follow such a pattern:
-  1. Create CubeMX project, set-up hardware configuration
-  2. Run stm32pio that automatically invoke CubeMX to generate the code, create PlatformIO project, patch an '.ini' file and so on
-  3. Work on the project in your editor, compile/upload/debug it
+  1. Create CubeMX project, set-up your hardware configuration
+  2. Run stm32pio that automatically invoke CubeMX to generate the code, create PlatformIO project, patch a 'platformio.ini' file and so on
+  3. Work on the project in your editor, compile/upload/debug etc.
   4. Edit the configuration in CubeMX when necessary, then run stm32pio to regenerate the code.
 
 Refer to Example section on more detailed steps.
 
-stm32pio will create an accessory file `cubemx-script`'` in your project directory that contains commands passed to CubeMX. You can safely delete it (it will be created again on the next run) or edit corresponding to your goals.
-
-Check `settings.py` to make sure that all user-specific parameters are valid. Run
+On the first run stm32pio will create a config file `stm32pio.ini`, syntax of which is similar to the `platformio.ini`. You can also create this config without any following operations by initializing the project:
 ```shell script
-$ python3 stm32pio.py --help
+$ stm32pio init -d path/to/project
 ```
-to see help.
+It may be useful to tweak some parameters before proceeding. The structure of the config is separated in two sections: `app` and `project`. Options of the first one is related to the global settings such as commands to invoke different instruments though they can be adjusted on the per-project base while the second section contains of project-related parameters. See the comments in the [`settings.py`](/stm32pio/settings.py) file for parameters description.
 
-
-## Installation
-Starting from v0.8 it is possible to install the utility to be able to run stm32pio from anywhere. Use
+You can always run
 ```shell script
-stm32pio-repo/ $   pip3 install .
-```
-command to launch the setup process. To uninstall run
-```shell script
-$ pip3 uninstall stm32pio
+$ python3 app.py --help
 ```
+to see help on available commands.
+
+You can also use stm32pio as a package and embed it in your own application. See [`app.py`](/stm32pio/app.py) to see how to implement this. Basically you need to import `stm32pio.lib` module (where the main `Stm32pio` class resides), set up a logger and you are good to go. If you need higher-level API similar to the CLI version use `main()` function in `app.py` passing the same CLI arguments to it.
 
 
 ## Example
@@ -74,26 +75,45 @@ $ pip3 uninstall stm32pio
 5. Run `platformio boards` (`pio boards`) or go to [boards](https://docs.platformio.org/en/latest/boards) to list all supported devices. Pick one and use its ID as a `-b` argument (for example, `nucleo_f031k6`)
 6. All done! You can now run
    ```shell script
-   $ python3 stm32pio.py new -d path/to/cubemx/project/ -b nucleo_f031k6 --start-editor=vscode --with-build
+   $ python3 app.py new -d path/to/cubemx/project/ -b nucleo_f031k6 --start-editor=code --with-build
    ```
-   to complete generation, start the Visual Studio Code editor with opened folder and compile the project (as an example, not required). Make sure you have all tools in PATH (`java` (or set its path in `settings.py`), `python`, editor). You can use shorter form if you are already located in the project directory (also using shebang alias):
+   to complete generation, start the Visual Studio Code editor with opened folder and compile the project (as an example, not required). Make sure you have all tools in PATH (`java` (or set its path in `stm32pio.ini`), `platformio`, `python`, editor). You can use shorter form if you are already located in the project directory (also using shebang alias):
    ```shell script
-   path/to/cubemx/project/ $   stm32pio.py new -b nucleo_f031k6
+   path/to/cubemx/project/ $   stm32pio new -b nucleo_f031k6
    ```
 7. If you will be in need to update hardware configuration in the future, make all necessary stuff in CubeMX and run `generate` command in a similar way:
    ```shell script
-   $ python3 stm32pio.py generate -d /path/to/cubemx/project
+   $ python3 app.py generate -d /path/to/cubemx/project
    ```
 8. To clean-up the folder and keep only the `.ioc` file run `clean` command
 
 
 ## Testing
-Since ver. 0.45 there are some unit-tests in file `stm32pio/tests/test.py` (based on the unittest module). Run
+Since ver. 0.45 there are some tests in file [`test.py`](/stm32pio/tests/test.py) (based on the unittest module). Run
 ```shell script
-stm32pio-repo/ $   python3 -m unittest discover -v
+stm32pio-repo/ $   python3 -m unittest -b -v
 ```
 or
 ```shell script
-stm32pio-repo/ $   python3 -m stm32pio.tests.test -v
+stm32pio-repo/ $   python3 -m stm32pio.tests.test -b -v
+```
+to test the app. It uses STM32F0 framework to generate and build a code from the test [`stm32pio-test-project.ioc`](/stm32pio-test-project/stm32pio-test-project.ioc) project file. Please make sure that the test project folder is clean (i.e. contains only an .ioc file) before running the test.
+
+For specific test suite or case you can use
+```shell script
+stm32pio-repo/ $   python3 -m unittest stm32pio.tests.test.TestIntegration -b -v
+stm32pio-repo/ $   python3 -m unittest stm32pio.tests.test.TestCLI.test_verbose -b -v
 ```
-to test the app. It uses STM32F0 framework to generate and build a code from the `stm32pio/tests/stm32pio-test-project/stm32pio-test-project.ioc` file. It's fine to fail an editor test as you not necessarily should have all the editors on your machine. CI is hard to implement for all target OSes during the requirement to have all tools (PlatformIO, Java, CubeMX, etc.) installed during the test. For example, ST doesn't even provide a direct link to CubeMX for downloading
+
+While testing was performed on different Python and OS versions, some older Windows versions had shown some 'glitches' and instability. [WinError 5] and others had appeared on such tests like `test_run_edtor` and on `tempfile` clean-up processes. So be ready to face off with them.
+
+CI is hard to implement for all target OSes during the requirement to have all the tools (PlatformIO, Java, CubeMX, etc.) installed during the test. For example, ST doesn't even provide a direct link to the CubeMX for downloading.
+
+
+## Restrictions
+  - The tool doesn't check for different parameters compatibility, e.g. CPU frequency, memory sizes and so on. It simply eases your workflow with these 2 programs (PlatformIO and STM32CubeMX) a little bit.
+  - CubeMX middlewares are not supported yet because it's hard to be prepared for every possible configuration. You need to manually adjust them to build appropriately. For example, FreeRTOS can be added via PlatformIO' `lib` feature or be directly compiled in its own directory using `lib_extra_dirs` option:
+    ```ini
+    lib_extra_dirs = Middlewares/Third_Party/FreeRTOS
+    ```
+    You also need to move all `.c`/`.h` files to the `src`/`include` folders respectively. See PlatformIO documentation for more information.

TODO.md

@@ -1,7 +1,34 @@
 # TODOs
+
  - [ ] Middleware support (FreeRTOS, etc.)
+ - [ ] Arduino framework support
  - [ ] Add more checks, for example when updating the project (`generate` command), check for boards matching and so on...
- - [ ] Do we need some sort of GUI? For example, drop the folder into small window (with checkboxes corresponding with CLI options) and get the output
- - [ ] remade as Class (constructor init(project_path))
- - [ ] test CLI (i.e. run stm32pio as subprocess)
- - [ ] upload to PyPI
+ - [x] Function annotations
+ - [ ] GUI. For example, drop the folder into small window (with checkboxes corresponding with CLI options) and get the output. At left is a list of recent projects
+ - [ ] GUI. Indicate the progress as states goes forward during the run (see `scratch.py`)
+ - [x] Remade as Class (constructor `__init__(project_path)`)
+ - [x] Config file for every project instead of the `settings.py` (but we still sill be storing the default parameters there)
+ - [x] Test CLI (integration testing)
+ - [x] Move test fixtures out of the 'tests' so we can use it for multiple projects (for example while testing CLI and GUI versions). Set up test folder for every single test so we make sure the .ioc file is always present and not deleted after failed test
+ - [ ] Upload to PyPI
+ - [x] `__main__`
+ - [x] Abort `--with-build` if no platformio.ini file is present
+ - [x] Rename `stm32pio.py` -> `app.py`
+ - [x] Rename `util.py` -> `lib.py` (maybe)
+ - [x] Do not require matching of the project folder and .ioc file names (use first .ioc file found)
+ - [x] Remove casts to string where we can use path-like objects
+ - [x] Settings string templates and multi line
+ - [x] Smart `start_editor` test (detect editors in system, maybe use unittest `skipIf` decorator)
+ - [x] `init` command
+ - [x] New argparse algo cause now we have config file
+ - [x] Update `.ioc` file
+ - [x] `str(path)` -> `path` were possible
+ - [x] Check `start_editor()` for different input
+ - [x] Test on Python 3.6 (pyenv)
+ - [ ] Test for `get_state()` (as sequence of states (see scratch.py))
+ - [x] Remake `get_state()` as property value (read-only getter with decorator)
+ - [ ] If the project path is a unique identifier of the project in our code maybe we can remake `Stm32pio` class as a subclass of `pathlib.Path` and then reference it like `self` and not `self.project_path`. It would be more consistent also, as now `project_path` is perceived like any other config parameter that somehow is appeared to exist outside of a config instance but then it will be a core identifier, a truly `self` value.
+ - [x] Try to invoke stm32pio as module (-m), from different paths...
+ - [ ] Logs format test (see prepared regular expressions)
+ - [ ] Some non-intrusive installation test (may be some sort of temp virtualenv...)
+ - [ ] We look for some snippets of strings in logs and output for the testing code but we hard-code them and this is not good

setup.py

@@ -1,30 +1,33 @@
 import setuptools
 
-from stm32pio.stm32pio import __version__
+import stm32pio.app
 
-with open("README.md", 'r') as fh:
-    long_description = fh.read()
+with open("README.md", 'r') as readme:
+    long_description = readme.read()
 
 setuptools.setup(
-    name="stm32pio",
-    version=__version__,
-    author="ussserrr",
-    author_email="[email protected]",
+    name='stm32pio',
+    version=stm32pio.app.__version__,
+    author='ussserrr',
+    author_email='[email protected]',
     description="Small cross-platform Python app that can create and update PlatformIO projects from STM32CubeMX .ioc "
                 "files.",
     long_description=long_description,
     long_description_content_type="text/markdown",
     url="https://github.com/ussserrr/stm32pio",
     packages=setuptools.find_packages(),
     classifiers=[
-        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3 :: Only",
         "License :: OSI Approved :: MIT License",
         "Operating System :: OS Independent",
+        "Development Status :: 5 - Production/Stable",
+        "Environment :: Console",
+        "Topic :: Software Development :: Embedded Systems"
     ],
     include_package_data=True,
     entry_points={
         'console_scripts': [
-            'stm32pio = stm32pio.stm32pio:main'
+            'stm32pio = stm32pio.app:main'
         ]
     }
 )

stm32pio/tests/stm32pio-test-project/stm32pio-test-project.ioc → stm32pio-test-project/stm32pio-test-project.ioc

@@ -19,8 +19,9 @@ Mcu.PinsNb=6
 Mcu.ThirdPartyNb=0
 Mcu.UserConstants=
 Mcu.UserName=STM32F031K6Tx
-MxCube.Version=5.3.0
-MxDb.Version=DB.5.0.30
+MxCube.Version=5.4.0
+MxDb.Version=DB.5.0.40
+NVIC.ForceEnableDMAVector=true
 NVIC.HardFault_IRQn=true\:0\:0\:false\:false\:true\:false\:false
 NVIC.NonMaskableInt_IRQn=true\:0\:0\:false\:false\:true\:false\:false
 NVIC.PendSV_IRQn=true\:0\:0\:false\:false\:true\:false\:false
@@ -67,7 +68,7 @@ ProjectManager.CustomerFirmwarePackage=
 ProjectManager.DefaultFWLocation=true
 ProjectManager.DeletePrevious=true
 ProjectManager.DeviceId=STM32F031K6Tx
-ProjectManager.FirmwarePackage=STM32Cube FW_F0 V1.10.1
+ProjectManager.FirmwarePackage=STM32Cube FW_F0 V1.11.0
 ProjectManager.FreePins=false
 ProjectManager.HalAssertFull=false
 ProjectManager.HeapSize=0x200
@@ -83,7 +84,7 @@ ProjectManager.ProjectName=stm32pio-test-project
 ProjectManager.StackSize=0x400
 ProjectManager.TargetToolchain=Other Toolchains (GPDSC)
 ProjectManager.ToolChainLocation=
-ProjectManager.UnderRoot=true
+ProjectManager.UnderRoot=false
 ProjectManager.functionlistsort=1-MX_GPIO_Init-GPIO-false-HAL-true,2-SystemClock_Config-RCC-false-HAL-false,3-MX_USART1_UART_Init-USART1-false-HAL-true
 RCC.CECFreq_Value=32786.88524590164
 RCC.FamilyName=M

stm32pio/__main__.py

@@ -0,0 +1,6 @@
+import sys
+
+import stm32pio.app
+
+if __name__ == '__main__':
+    sys.exit(stm32pio.app.main())