Setting up a productive development environment...?

[gtwundke]

Hi all,
I'm currently thinking about doing some BW development. I'm a software engineer by trade, and figure I could fix some personal niggles, as well as perhaps contributing to the overall volunteer effort.

Having read a decent number of posts on here, I've got myself a shiny new Ubuntu VM and can build the easy-ui-4 repository with the latest SDK; not that I necessarily expect to delve into the C++ too much, but it's nice to know that it all builds as expected.

However, I'm curious as to what IDEs people are using (especially for Python development), and how they might set them up to work productively. My preference would be to use PyCharm if possible (having used IntelliJ for many years now), but that may not be ideal.

My two biggest issues at the moment are:

1. The top-level Python code lives in the root of the repository, but the sub-packages are all in lib/python, which seems like an odd way to lay things out (although I gather this is from the upstream repository, and not really something we can change). This makes viewing/changing those top-level modules difficult due to errors and no easy navigation to imported modules. How do you guys get around this? Symlinks? Some funky IDE setup?
2. How do you setup the IDE (or Python) such that it knows which site-packages are available in the SDK? I'll admit that this is a blind spot of mine, as the majority of the Python work I've done hasn't had to deal with virtualenv etc.

I'm sure there will be more questions, but this seems like a good start.

Tim.

[prl]

I'm an old-timer (I started using Unix in about 1979) and I don't use an IDE for development at all. I'm aware that that makes me a dinosaur

For development, I run a bunch of xterms inside MacOS XQuartz, and I edit with the sam editor (originally developed to run on the AT&T Blit/DMD 5620 smart terminal). Sam is also fairly old school (late 1980s), but has fairly powerful regular expression matching (including multi-line REs).

When I'm working on code, I simply cd to .../easy-ui-4/lib/python and use relative path names from there.

I use a combination of command-line git and the Bitbucket browser UI for interaction with the repositories.

That suffices for all my Python development. I have a couple of scripts that I made to install lists of files from the Mac to the PVR and uninstall them when I'm done.

I compile enigma2 using the build environment that peteru makes available on Ubuntu 16.04.5 LTS inside a VirtualBox VM.

[adoxa]

I don't use an IDE either, just TDE in the Windows 7 console (with easy-ui-4 as cwd). I have a batch file to compile the Python files using the PVR directory and a script to compile just the enigma2 binary (using Ubuntu 16.04 in VirtualBox). Patches use robocopy to copy to USB in the PVR structure, which then use a script to copy (cp -R) to the PVR (no network access). I use Git for Windows, in its bash command line.

[Star6key]

...
Sam is also fairly old school (late 1908s), but has fairly powerful regular expression matching (including multi-line REs).
...

I would think that it would be more like black magic... in that day and age.

[prl]

...
Sam is also fairly old school (late 1908s), but has fairly powerful regular expression matching (including multi-line REs).
...

I would think that it would be more like black magic... in that day and age.

I thought I'd corrected that. Fixed now.

[gtwundke]

I'm an old-timer (I started using Unix in about 1979) and I don't use an IDE for development at all. I'm aware that that makes me a dinosaur

Heh, nothing wrong with being a dinosaur if it works for you! I personally can't fathom how you can be so productive without an IDE (I'm soft, I know!).

That suffices for all my Python development. I have a couple of scripts that I made to install lists of files from the Mac to the PVR and uninstall them when I'm done.

What's the chance of getting a copy of those scripts? I'm happy to create my own, but it's always nice to have a starting point if possible.

[gtwundke]

I don't use an IDE either, just TDE in the Windows 7 console (with easy-ui-4 as cwd). I have a batch file to compile the Python files using the PVR directory and a script to compile just the enigma2 binary (using Ubuntu 16.04 in VirtualBox). Patches use robocopy to copy to USB in the PVR structure, which then use a script to copy (cp -R) to the PVR (no network access). I use Git for Windows, in its bash command line.

Wow, I can't believe there's another person not using an IDE!

What's the possibility of getting a copy of the batch file that you use to compile the Python? As per my second question, I assume it sets the Python environment up to reference the site-packages in the SDK? More info on this would be helpful.

Tim.

[IanSav]

Hi Tim,

Sorry to disappoint you but I also don't use an IDE. I use Win32Pad and now Notepad++ as the text editors on Windows 10. If actually working directly on a PVR I use VI as the editor.

I have a number of the repositories cloned to my NAS. I cross mount, via SMB, my Windows desktops, a Beyonwiz and an OpenViX PVR (and occasionally OpenPLi PVR) to run various scripts, Linux and Windows, that manage the various development efforts on the different platforms. I also use SyncBack to synchronise the various development platforms.

I use the GitHub and BitBucket web clients and well as the SourceTree Windows GIT client to manage the repository interactions.

Regards,
Ian.

[adoxa]

What's the possibility of getting a copy of the batch file that you use to compile the Python? As per my second question, I assume it sets the Python environment up to reference the site-packages in the SDK? More info on this would be helpful.

Here's the batch. I don't do anything with site-packages.

Code: Select all

@echo off
SetLocal EnableDelayedExpansion
set python=c:\language\python27\python.exe
for %%P in (%*) do (
  set DDIR=%%~pP
  set DDIR=!DDIR:\=/!
  set DDIR=!DDIR:/Projects/beyonwiz/easy-ui-4/lib/python=/usr/lib/enigma2/python!
  %python% -O -m compileall -d !DDIR! %%P
)

I have that in the beyonwiz directory, but I'm normally in the easy-ui-4 subdirectory, so I generally compile like ..\compile lib\python\Screens\EpgSelection.py. Then it's robocopy lib\python e:\working\usr\lib\enigma2\python *.pyo /s /mov /ndl /njh /njs /xx |sed s/%CD:\=\\%\\lib\\python\\// to put the compiled files onto USB. Then I run a script from File Commander to install them on the box.

[peteru]

All my development hosts are Linux machines. Mainly Gentoo ~amd64, although I also have Gentoo x86. No Windows systems in sight.

Eclipse IDE for C/C++ Developers
Version: Oxygen.3a Release (4.7.3a)
Build id: 20180405-1200

with

PyDev - Python Development environment.
Version: 6.5.0.201809011628

I also have the Yocto project plugin installed, plus a few more, but those are for development on other hardware platforms.

The projects required a fair bit of setting up, including pointing to paths in the SDK sysroot, but I have full cross-references going in Python and C++ code. Some of the SWIG generated glue can still throw a curve ball here and there, but it works fairly well.

I had a go at PyCharm and it failed badly. So did Visual Code. I only had a quick go with Atom and it didn't look promising either.

My requirements are probably well above most contributors, since I deal with the whole gamut, ranging from build system metadata, through kernel, drivers, libraries, system scripts and all the way to CSS for web pages. Eclipse does a pretty good job on most of it. It does require a lot of memory. I have the Eclipse Java VM set to use 4GB of RAM. When doing a full firmware build, systems with 16GB to 32GB RAM work for me. Then again, I'll routinely have several project open, including a couple of version of the Linux kernel, enigma2, a number of plugins, GStreamer, Samba, etc. It all adds up.

I use rsync or Dolphin with fish:// or scp:// to push files to the targets. Git-gui, gitk and git command line, together with Bitbucket/Github web interface for git work. I tried a number of Git GUI clients, but in the end decided that none of them make the job any easier. Every now and then, I just use vim or kwrite for quick edits.

Trickier task often get a line or two in a "cheat sheet file" to remind myself later. Repetitive things are usually scripted or added to GNU Makefiles.

Between development and feeds I use about 12 systems, no virtual machines or containers. I don't particularly want the builds to take any longer, it's bad enough to have to wait several hours as it is now. It's also likely to eat more memory. Given so many systems are involved, ssh with properly distributed keys is essential.

For debugging, I have a bunch of USB-serial converters and run minicom. It's configured to capture everything to log files that are stored permanently. I have several years worth of serial port logs from some systems.

Lots of storage is mandatory. At the moment I am using about 12TB, but I had to do some major cleaning up last week because I ran out of space. The NAS is out of drive bays, as is the laptop with three SSDs. My workstation could fit another drive, but then I won't have the space to back it up.

I'm sure you can still get decent results with a lot less, but if you want one IDE that groks everything, Eclipse is the only thing that I have found. At times, it is hard work setting it up, but it's worth it when you can just click or hover over code and get the relevant context sensitive info you need. The only thing missing is cross-debugging. I tried to get it going, but the MIPS remote client isn't capable enough. I have not tried with the ARM based machines.

[gtwundke]

Sorry to disappoint you but I also don't use an IDE. I use Win32Pad and now Notepad++ as the text editors on Windows 10. If actually working directly on a PVR I use VI as the editor.

Another one! I fully expect someone to chime in next saying they use edlin! I'm glad you've at least moved to Notepad++ though. VI is my go-to quick editor on Linux too.

I have a number of the repositories cloned to my NAS. I cross mount, via SMB, my Windows desktops, a Beyonwiz and an OpenViX PVR (and occasionally OpenPLi PVR) to run various scripts, Linux and Windows, that manage the various development efforts on the different platforms. I also use SyncBack to synchronise the various development platforms.

Ahh, yes. Mounting the Beyonwiz is a good idea that I hadn't thought of.

I use the GitHub and BitBucket web clients and well as the SourceTree Windows GIT client to manage the repository interactions.

I know that Atlassian made an attempt to speed up SourceTree a few years ago, but it never provided enough benefit for me, so I dropped it.

It might be worth checking out GitKraken if you're still finding SourceTree a bit sluggish, as it was built to be a super-fast Git client. I haven't used it myself, but a lot of people at work swear by it. Plus it's fully cross platform.

Tim.

[gtwundke]

Here's the batch. I don't do anything with site-packages.

Code: Select all

@echo off
SetLocal EnableDelayedExpansion
set python=c:\language\python27\python.exe
for %%P in (%*) do (
  set DDIR=%%~pP
  set DDIR=!DDIR:\=/!
  set DDIR=!DDIR:/Projects/beyonwiz/easy-ui-4/lib/python=/usr/lib/enigma2/python!
  %python% -O -m compileall -d !DDIR! %%P
)

Ok, that makes sense. In my head I thought compilation would check that imports were available, but it obviously just spits out bytecode regardless of whether it will work when deployed, as is Python's general philosophy given its dynamic nature.

Thanks for that!

Tim.

[prl]

Here's the batch. I don't do anything with site-packages.

Nice, now that I've worked out what it does. I've always wondered why people complain about Unix shell being cryptic.

I'll post a bash translation when I've done it if anyone's interested.

[gtwundke]

Wow, thanks for all of the information Peter! Questions below

Eclipse IDE for C/C++ Developers
PyDev - Python Development environment.

Looks like I'm going to have to renounce my vow to never use Eclipse . I actively avoid it for Java development (my main gig) as I've just never been comfortable with the way it works (and had some bad experiences early on that soured my perception of it), whereas IntelliJ IDEA just tended to work (mostly ).

The projects required a fair bit of setting up, including pointing to paths in the SDK sysroot, but I have full cross-references going in Python and C++ code. Some of the SWIG generated glue can still throw a curve ball here and there, but it works fairly well.

Nice! You may guess what's coming next, given my replies above, but what are the chances of you exporting your project/library configurations (I'm not 100% sure of the Eclipse terminology here) for me to have a look at? I'm happy to deal with translating paths etc, which will obviously be specific to your configuration. If it's all too much of a pain, that's fine, but I figure I may as well ask .

I had a go at PyCharm and it failed badly. So did Visual Code. I only had a quick go with Atom and it didn't look promising either.

Do you happen to remember what the issue(s) with PyCharm were?

For debugging, I have a bunch of USB-serial converters and run minicom. It's configured to capture everything to log files that are stored permanently. I have several years worth of serial port logs from some systems.

I'd be interested in hearing more about how this all works, as I'm not familiar with what's being logged via the serial port. Also, the U4 doesn't have an external serial port, so is this still possible?

Lots of storage is mandatory. At the moment I am using about 12TB, but I had to do some major cleaning up last week because I ran out of space. The NAS is out of drive bays, as is the laptop with three SSDs. My workstation could fit another drive, but then I won't have the space to back it up.

I have an 18TB NAS sitting here with plenty of space left on it, so no real worries there at the moment.

I'm sure you can still get decent results with a lot less, but if you want one IDE that groks everything, Eclipse is the only thing that I have found. At times, it is hard work setting it up, but it's worth it when you can just click or hover over code and get the relevant context sensitive info you need.

The other guys have proved to me that an IDE obviously isn't a requirement, but I much prefer to work in an easily-navigable environment.

Tim.

[prl]

That suffices for all my Python development. I have a couple of scripts that I made to install lists of files from the Mac to the PVR and uninstall them when I'm done.

What's the chance of getting a copy of those scripts? I'm happy to create my own, but it's always nice to have a starting point if possible.

No problem. They have some relative paths of other components (like skins) as variables that you'd have to change, but that shouldn't be a problem - but they're bash scripts not Windows batch files if you're using a Windows environment.

My scripts also use ssh (or rather scp) to push the patch scripts that it generates to the PVR. That could be relatively easily changed to copy o a mount of the PVR's HDD on the development machine.

Adoxa has an alternative (and in some ways better) setup for doing the same thing.

[gtwundke]

No problem. They have some relative paths of other components (like skins) as variables that you'd have to change, but that shouldn't be a problem - but they're bash scripts not Windows batch files if you're using a Windows environment.

Bash scripts are fine; I'm comfortable with both. I'll probably develop on Linux for the moment anyway.

Tim.

[MrQuade]

FYI, not that it has been brought up yet, but my experiments with building the code using Windows Subsystem for Linux (WSL) have not been overly successful. Part of that is due to my inexperience no doubt, but I think a big part of it is the evolving nature of the WSL.

I did have it in a state where I could compile easy-ui-4 at one point, but ongoing changes to the system on Microsoft's end seem to keep breaking things. It may be usable, but probably not a stable enough platform to rely upon yet. Which is a shame.

[gtwundke]

FYI, not that it has been brought up yet, but my experiments with building the code using Windows Subsystem for Linux (WSL) have not been overly successful. Part of that is due to my inexperience no doubt, but I think a big part of it is the evolving nature of the WSL.

I did have it in a state where I could compile easy-ui-4 at one point, but ongoing changes to the system on Microsoft's end seem to keep breaking things. It may be usable, but probably not a stable enough platform to rely upon yet. Which is a shame.

I was actually wondering about that. I saw your other post about getting it up and running, and did briefly consider it myself, but figured sticking with something that was known to work was probably best at this early stage. It's a pity that it's not stable though.

Tim.