After several months working fine, the datalogger v1.0, described in the project has been a success, without any relevant incident, but it has always required to have a computer started. It has been power outages and the computer has restarted without problems but not the serial port attached to the Arduino, and some sensor samples have been lost during these periods. So I decided to design a more robust datalogger, following some ideas already present in jardiNet.
Summarizing these are the new features implemented in the datalogger v2.0:
Just two comments on this: Arduino Uno has only 32KB of memory and the Mega 256KB, so to implement all features in Arduino Uno it is necessary a very good developer, as my code needs actually about 36KB. I'm not such developer. The other point is that Arduino Uno has only one serial port and the Mega has 4 ports and this has been definitive in order to get a more versatile system: I power the Arduino using the USB connector associated with serial port 0 and I use serial port 3 to connect with the computer. Note also that serial port 0 is also used by the XBee.
So I need to find some shield with SD and XBee support. I've tested two shields: 'Arduino Wireless SD Shield' i 'Adafruit Assembled Data Logging shield for Arduino'. So finally I used the shield from Adafruit because it has the Arduino pinout 1.0, so it has additional SCL and SDA pins next to AREF pin, which are needed by the LCD shield.
But there are also advantages using the Arduino shield: it is possible to use an updated version of the sdfat library, instead of using the standard SD library that comes with the Arduino IDE. And this is not a trivial question as the standard SD library does not support usual functions like a file rename, so I've needed to code my own function to rename files in the hard way (create a copy with the new name and delete the original one). Also the Arduino shield has a switch that simplifies a lot the development as it removes the need to physically unplug the XBee to upload Arduino code, but this can be addressed just adding a switch and few wires to connect the XBee to the Arduino, as shown in next pictures.
It is necessary that Arduino could generate time references when receiving sample data. In v1.0, the Mac was generating timestamps at serial packet reception from Arduino, but if we want an standalone datalogger it has to generate the timestamps. I've been looking for a component easily attachable to the Mega: for Arduino Wireless SD shield I've tested successfully an independent RTC, Adafruit 264, but as I finally used the Adafruit shield which already has an RTC I simplified this problem. In other way, just note that using an RTC from Adafruit means to install and use the RTClib in the Arduino IDE. Finally, the Adafruit shield requires a custom SD library. Both Adafruit libraries can be obtained at learn.adafruit.com
I've replaced the Objective-C code in arduinoToSQLite by a python script. I've replaced about 1500 Objective-C lines (sample code used), plus other resources like the user interface design objects, by a 150 python lines. Python is really powerful for data extraction and processing. I've also need to say that I've lost a very easy to use user interface, replaced by a simple configuration file, but it work!!
For sure is not really needed at all as it doesn't implement anything critical, but it helps a lot to know the datalogger status, adjust time or just see the last sample read from sensor. I've choosen this one: Adafruit RGB LCD Shield 16x2. But I've changed a little the assembly process, following this post in Adafruit forum, in order to manage the keyboard interrupts in the Arduino. It is only necessary to add a wire (soldered) in the LCD display (MCP23017 pin 20) and modify the Adafruit libraries provided, as explained in documentation.
I said it before: the Arduino Mega has four serial ports and I need two of them. Arduino has only one, but when XBee receives data and send it to Arduino, the serial port 0 is still connected to the USB-to-serial converter, so it is possible to send data to the computer at same time it's received from sensor. Unfortunately the Arduino cannot be uploaded with new code with this setup. So I use serial port 0 for Arduino-XBee communications and to power the beast and I've added an USB-to-serial shield to serial port 3, Adafruit FTDI friend, to mantain computer connectivity.
Even when the basic functionallity of the datalogger is already described in jardiNet v1.0 I will briefly describe it again.
The new datalogger is composed by: Arduino Mega 2560, which supports the rest of shields. The first shield is the 'Adafruit Data Logging shield for Arduino (1141)' and on top of it the display 'RGB LCD Shield Kit w/ 16x2 Character Display (714)' also from Adafruit. The 'Adafruit FTDI Friend (284)' is just wired to the Mega:
You already know that Arduino programming requires the serial port 0 and this produces a conflict with the XBee which also use it. So if we have the XBee plugged in the shield is not possible to program the Arduino and it has to be extracted to do it. This is not so efficient because at development time is too frequent to have to upload code, so it is possible to damage a pin in the XBee or the shield. So, during development I mounted the XBee in a protoboard on top of an 'SparkFun XBee Explorer (WRL-09132)' and I wired a switch between the DOUT pin in the XBee and the corresponding pin in the shield, look at the picture to understand how:
During tests and development I've used an XBee mounted on top of an XBee Explorer, which sent sample data every minute to simulate a real sensor radio. Obviously the data collected was only noise but it worked to test the set of components. Finally the completed datalogger has this look:
Tables below summarize all connections used between the Arduino Mega and the shields:
|Arduino Mega 2560 pin||Adafruit Data Logging shield for Arduino (1141)|
|3.3V||1 Vcc XBee|
|GND||10 GND XBee|
|0 RX||2 DOUT, TX XBee|
|1 TX||3 DIN, RX XBee|
|10||CS (SS) pin as OUTPUT in Arduino|
|Arduino Mega 2560 pin||Adafruit 264 (FTDI Friend, serial to USB)|
|GND (costat pin 53)||GND|
|Arduino Mega 2560 pin||Adafruit 714 (RGB LCD shield 16x2)|
|20 SDA (posició Arduino UNO)||SDA|
|21 SCL (posició Arduino UNO)||SCL|
Date flow is a little bit different than in version 1.0. Now, the datalogger converts the readed values in XBee analogic inputs to millivolts, in this way is transparent in the data flow (before I converted data before to be saved in db). So in db we just save the values in mVolts provided by sensor and it's the web service coded in python, finally, who reads the database and applies the formula to convert this value in mVolts to VWC%, so the final value is obtained at query time.
I prefer to have the latest calculation in the web page and mantain in db values in mVolts, because now I'm using the standard
formula provided by sensor manufacturer but it is recommeded to calibrate the sensor to have a better precision. If I do it I
just have to change the web service code, without any kind of conversion.
The most important thing is the replacement of Objective-C code by python which it's platfom independent. MacOSX is version 10.8.5 an python 2.7.2. Is necessary to add the serial module which can be found: pySerial
Arduino code has also changed a lot but not the portion of code that manages the XBee. Code is now complex as the Arduino has to manage interruptions to respond to the keyboard in display. The Arduino does not have a multi-threaded architecture to solve this issue in a clever way, so code is not easy to understand, but it works. !!
The code now is always storing sensor data in the SD card and if for any reason the connection is lost with computer all samples will be saved with the prefix 'PENDENT:'. In this way, when the Arduino recovers the connection to the computer, the data file is read and data is sent to the db. File is moved to another subdirectory in the SD card and a new data file is created
This flow chart shows, in a simplified version, how the code works in the Arduino:
handshake() function is executed inside loop() and it manages the connection with the computer, in this way the Arduino knows if the data must be sent to db or it is only to be saved in SD card. We have to establish a compromise between the frequency of handshake() runs and the time that loop() uses to receive XBee packets. If we execute the handshake() too frequently there is the risk to loose sensor data packets. In the actual version, handshake() runs every 5 minutes and most of the loop cycles are used to run function xbee.readPacket(XBEE_TIMEOUT) which allows to read the sensor data and process it.
Finally just to comment that the web service code, already in python has not changed.
Attached is the code in use: jardiNet_v2.0.zip. You can freely use it but if you improve it, please, explain it to me to see if I can also improve it.
The LCD display has several buttons that allow a simple interaction with the Arduino, but I've needed to change the Adafruit board a little and also the library provided, in order to implement keyboard interruptions in the Arduino code. The following table shows the functions of each key:
|Select mode||Press to select: Set date & time, Save time, XBee mode|
|Up||Set date & time: increase|
|Down||XBee mode: current time (no DST support), Set date & time: decrease|
|<-- (Left)||XBee mode: last sensor values, Set date & time: move cursor|
|--> (Right)-->||XBee mode: status SD card and computer connection, Set date & time: move cursor|
|Reset||Restart the Arduino|
|Brightness||Adjust display brightness, screwdriver required|
1.0 version is now history and in 2.0 only the datalogger has been improved, but now I'm thinking in 3.0. This is the whish list as it never ends !!
I would like to integrate all sensor components in a unique circuit. Adafruit 390 is a versatile circuit with a lot of applications but in this case I would like to change it to have stabilized voltage output of 3.3Volts and also the XBee sockets. It could be also interesting to provide enough space to integrate other sensors. Reference documentation is available in github and it's a good starting point. Any help here ??
The 2.0 datalogger 2.0 is a great improvement but it's still required a computer to store data and publish it in internet, in real time. So a 3.0 datalogger has to be completely independent with no computer at all. I'm evaluating these new hybrid Arduinos: Arduino Yún or Arduino Tre, both are Arduinos with a processor that runs a customized Linux version, with enough RAM memory (64MB in Yún and 512MB in Tre), and with Ethernet/SD. The server side of the Arduino has to be able to run SQLite, to store the sensor data, and Cherrypy to run the web service. Yún is now available but for Tre we will need to wait until 2014, time for a break....
To complete the project is necessary to build a remote controlled faucet. In this way, the Arduino can order to water the garden if the water levels are so low. Design will be based on the remote sensor already in place. Also a further enhancement could be to allow to water from an internet application, why not ? But I will need to think about security in maslestorres.cat.