RELEASE: Cordial v1.4.2

We are very happy to announce this release of cordial v1.4.2

Note: v1.4.2 is a small set of fixes for issues discovered by users in testing v1.4.1 - please keep the reports coming.

Apart from many bug fixes to the geneos program there has been work done on improving support for remote host management (you can now use an encoded password, if you can’t use key files or an agent), along with the required support for Geneos Gateway compatible AES key files, especially for opaquing credentials in configurations, and improvements to the install and update commands to make them behave more predictably.

Other changes include new APIs in the packages and support for more configuration options including option expressions in configuration values and more.

Please update and let us know how you get on.

Change Log

Version v1.4.2 - 2022/12/21

  • Fixes

    • tools/geneos: fix update to only optional restart (-R) the component type given
    • tools/geneos: check RHEL8 download in a case independent way - fixes remotes
    • tools/geneos: create user config directory for remote hosts in case of old location for main config
    • tools/geneos: install should error out is passed @host instead of -H host
    • tools/geneos: ssh known hosts handling improved (for mixed IP / hostnames)
    • tools/geneos: remote hosts with IP names are now renamed A-B-C-D to avoid issues with viper names

Version v1.4.1 - 2022/12/19

  • Fixes

    • tools/geneos: check return from user.Current() as it can fail (but shouldn’t)
    • tools/geneos: numerous fixes for logic around handling of remote hosts
    • tools/geneos: fix remote host naming to be work with capitalisations
    • tools/geneos: actually load SSH private key files, if available
    • tools/geneos: re-order SSH HostKeyAlgorithms so that, bizarrely, IP based remotes work
    • tools/geneos: better handling of instance config aliases when writing config files
    • tools/geneos: fixes to unset to ignore values that may be passed in with keys to unset
    • tools/geneos: refactor CopyInstance() to preserve ports, other details
    • build: create static executables, using alpine, and a centos 7 compatible libemail.so
    • tools/geneos: add the beginnings of support for YAML instance config files. not enabled yet.
    • tools/geneos: fix crash when importing to common directories of components without the component name
    • tools/geneos: fix fileagent support by adding implicit imports with side-effects for all component packages
    • tools/geneos: skip failed permissions on /proc/*/fd - let ‘ps’ work for restricted processes
    • tools/geneos: fix update-during-install support, add --force flag for this too
    • tools/geneos: fix logic to match latest packages when major number changes
  • Changes

    • tools/geneos: clean-up various comments, refactor methods, add license/copyright notices to many files
    • pkg/config: Add an options expr prefix to expansion items which supports [GitHub - maja42/goval: Expression evaluation in golang] syntax
    • pkg/config: API change: Add options to the config expansion functions rather than just lookup maps
    • tools/geneos: add SSH password support for remote hosts
    • tools/geneos: support embedded SSH passwords in hosts config, using new ‘set host’ sub-command
    • tools/geneos: support additional SSH private key files per host via ‘set host sshkeys=X,Y’ sub-command
    • tools/geneos: begin implementation of support for YAML config files via ‘configtype’ user setting
    • pkg/geneos: add EnvironmentRef and fix periodStartTime attribute
  • Other

    • tools/geneos: ongoing documentation and command help usage updates
    • tools/geneos: update README.md with more information about instance configuration files and their values (@gvastel)

Version v1.3.2 - 2022/11/02

  • Fixes

    • tools/geneos: fix running as root (or via sudo) and creation of config directories and file ownerships
    • tools/geneos: fix creation of full user config directories when running ‘set user’

Version v1.3.1 - 2022/11/01

  • Fixes

    • tools/geneos: chown files and directories creates when run as root
    • tools/geneos: ensure plain ‘init’ creates all components dirs

Version v1.3.0 - 2022/10/25

Changes

  • PagerDuty integration
  • Merged ServiceNow integration, single binary build
  • tools/geneos: add instance protection against stop (and related) or delete commands
  • tools/geneos: support legacy command through emulating *ctl named commands
  • tools/geneos: allow remote operations without local directories

Fixes

  • tools/geneos: fix logic around creating user default AES keyfiles and directory permissions
  • tools/geneos: round certificate expiry to midnight
  • tools/geneos: round tls remaining column to seconds correctly
  • tools/geneos: fix webserver command build typo. now webserver starts correctly
  • libemail: fix default _SUBJECT handling
  • tools/geneos: split over complex ‘init’ command into sub-commands
  • updated command usage information and reordered various internal function calls
  • tools/geneos: add password verify to aes encode and a --once flag to override
  • tools/geneos: add local JoinSlash and Dir to use Linux paths on Windows builds
  • tools/geneos: fix ssh-agent support on windows
  • tools/geneos: build on windows
  • integrations: Add PagerDuty integration
  • Integrations: Merge ServiceNow binaries into one
  • tools/geneos: change internal remote Stat() API

Notes for previous release, v1.2.1

There have been many changes since the original v1.0.0 release in June 2022. Please see the CHANGELOG.md file for more information.

The highlights include:

  • Lots of geneos program improvements, including numerous bug fixes and new features like AES encoding support
  • The pkgs get many improvements, the XML-RPC packages have been refactored to work in a real-world application
    • New config package to add a variety of features over the top of viper
    • New geneos package to aid in the construction of XML configurations for Gateway and Netprobe. This is work in progress.
    • New commands package to provide the start of support for REST API Commands to the Gateway. This is work in progress.
    • New xpath package to work with the above and also the base for the snapshot command below. This is also work in progress.
    • New cordial package that initially carries a version constant and a general purpose LogInit() function.
    • New process package, providing a way to Daemon()ise callers on both Linux and Windows.
  • The Servicenow integration has clean-up work under the hood, adding better configuration file support using the new package(s) above
  • Use of zerolog for logging in most cases throughout. The existing logger package will be deprecated in the next release

Individual binaries for Linux/AMD64 and example configurations can be found attached to the release, as well as tar.gz and zip files containing the same files for convenience.

Work continues and we appreciate any feedback, suggestions or bug reports you have. For bugs please either raise an Issue in github or, for more general things, post here.

We welcome feedback and contributions.


cordial

Cordial comprises a set of interrelated programs, integrations, libraries and packages for Geneos, principally written in Go.

  • geneos Manager Program
  • ServiceNow Incident Integration
  • Enhanced libemail
  • Go Packages
    • commands
      Geneos Gateway REST API Commands
    • config
      Configuration file support, based on viper with local extensions
    • geneos
      Automated Geneos XML configuration file generations based on Go data structures
    • process
      Process management utilities.
    • plugins, samplers, streams, xmlrpc
      Geneos API plugin XML-RPC support
    • xpath
      Geneos XPath handling

Version v1.2.1-rc3 - 2022/10/07

Fixes

  • geneos command fixes:

    • Fixed init download credential handling
    • Fixes JSON output formay from ls commands
    • Local-only installs now work again (including default “latest” support)
  • Security

    • Updated Labstack Echo to 4.9.0 to address security advisory CVE-2022-40083. To best of our knowledge this particular set of features was never used in this package.

Additional features and improvements

  • geneos command improvements:

    • Added --raw to show to not output expanded configuration values
    • Many improvements and changes to the new aes sub-commands. Please see documentation for details
    • Removed built-in opaquing of credentials in output in favour of new ${enc:...} support
  • libemail.so gets direct passwords back, with ExpandString support. See documentation for details

  • General package improvements

    • Enhanced OpenLocalFileOrURL to support ~/ paths
    • Enhanced ExpandString to support direct file paths and updates package docs further

Nice. I am using geneos tool since 2013. Its awesome. It will be great if this is available in reasonable price in order to compete other available monitoring tools in the market.

Hi Pinku, thanks for the positive comment. This specific tool set I have announced is not Geneos itself but tool(s) to help manage large, multi-component deployments that should improve ease of use over existing shell scripts etc.

Great news and add to the forum, thanks a lot @pgalbavy :+1:

Hello Peter,

It sounds great. We would need to use the SECURE/TLS extension of SMTP and ITRS forwarded me to this place. We would need to use libmail library to send secured mails. Thank you very much for your contribution :slight_smile:

Regards

If you visit the repo page in the first post you will see the ‘assets’ below the text - libemail.so is a directly downloadable file that can be used instead of the standard release version and will automatically enable TLS if connecting to an SMTP server port that uses it (the ‘default’ setting for SMTP_TLS).

If you are already using the standard libemail.so then you will have to put this file elsewhere and refer to it using a full path name, otherwise you can also consider overwriting it in place (but the gateway may need a restart if it ever loaded the old one).

Any issues, please post here!

Hello Peter,

Thank you for your help, I have been trying to send the mails today but unfortunately I failed on my mission, getting always the undesired RC in the log:
2022-07-26 17:48:02.794+0200 INFO: ACTION API Thread 139731673134848 return = 1 for libemail.so:SendMail(33

This is how I set the config:


<action name="Monitoring mail TLS">
    <sharedLibrary>
        <libraryFile>libemail.so</libraryFile>
        <functionName>SendMail</functionName>
        <staticParameters>
            <staticParameter>
                <name>_SMTP_SERVER</name>
                <value>
                    <data>smtp.net.intra</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_FROM</name>
                <value>
                    <data>sender@domain.com</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_SUBJECT</name>
                <value>
                    <data> %(_GATEWAY) - %(_MANAGED_ENTITY) %(_DATAVIEW) %(_VARIABLE) = %(_VALUE)</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_FORMAT</name>
                <value>
                    <data>***This is an auto-generated message from ITRS Gateway %(_GATEWAY)***
-----------------------------------------------------------------
Triggered at value:  %(_VALUE)
-----------------------------------------------------------------
</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_TO</name>
                <value>
                    <data>samuel.echevarria@domain.com</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_SMTP_USERNAME</name>
                <value>
                    <data>sender@domain.com</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_SMTP_PASSWORD_FILE</name>
                <value>
                    <data>mailpassword</data>
                </value>
            </staticParameter>
            <staticParameter>
                <name>_SMTP_TLS</name>
                <value>
                    <data>default</data>
                </value>
            </staticParameter>
        </staticParameters>
    </sharedLibrary>
</action>
****

If you have other configuration please let me know if I am missing a parameter.

Thank you

Regards

I do not see an SMTP_PORT in that config - for SSL/TLS you normally need to use a port other than 25 (the default). There is either the “msa” port (587) or the STARTTLS supporting port 465 (I think). Best check with your admins and/or test connectivity by hand first.

Also check the stderr output of the gateway, maybe gateway.txt if you use gatewayctl to run the gateway - it may contain more info. The challenge with writing code that makes debugging this stuff without spamming normal logs.

Also, I suggest ensuring you are calling the correct libemail.so file by using the full, absolute path in the libraryFile field.

Hello Peter,

Thank you for your feedback. May I ask you if you have an example XML of the action? Maybe I can use the same that works for you to discard other potential issues.

Thank you in advance

This one works for me (filling in the vars as appropriate):


<action name="test email">
	<sharedLibrary>
		<libraryFile>/home/peter/geneos/gateway/gateway_shared/libemail.so</libraryFile>
		<functionName>SendMail</functionName>
		<staticParameters>
			<staticParameter>
				<name>_SMTP_SERVER</name>
				<value>
					<data>smtp.gmail.com</data>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_SMTP_PORT</name>
				<value>
					<data>587</data>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_SMTP_USERNAME</name>
				<value>
					<var ref="username"/>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_SMTP_PASSWORD_FILE</name>
				<value>
					<var ref="password_file"/>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_FORMAT</name>
				<value>
					<data>&quot;Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Vivamus arcu felis bibendum ut tristique et egestas quis. Consequat nisl vel pretium lectus quam id leo in. Faucibus interdum posuere lorem ipsum dolor sit amet. Eu feugiat pretium nibh ipsum consequat nisl. Vitae tempus quam pellentesque nec. Consectetur purus ut faucibus pulvinar. Morbi tristique senectus et netus et malesuada fames. Sem viverra aliquet eget sit amet tellus cras. Pretium viverra suspendisse potenti nullam ac tortor vitae purus faucibus. Montes nascetur ridiculus mus mauris. Vulputate dignissim suspendisse in est ante in nibh mauris. Parturient montes nascetur ridiculus mus.

Fames ac turpis egestas maecenas pharetra. Aliquam sem fringilla ut morbi tincidunt augue interdum velit euismod. Libero enim sed faucibus turpis in eu. Proin sagittis nisl rhoncus mattis rhoncus urna neque viverra. Molestie at elementum eu facilisis sed odio morbi quis. Et malesuada fames ac turpis egestas integer eget aliquet nibh. Sit amet mattis vulputate enim nulla aliquet porttitor. Mi ipsum faucibus vitae aliquet nec ullamcorper. Dui faucibus in ornare quam viverra. Placerat vestibulum lectus mauris ultrices eros in cursus turpis massa. Odio euismod lacinia at quis.

Pellentesque adipiscing commodo elit at. Eget arcu dictum varius duis at. Nunc sed augue lacus viverra vitae congue eu consequat ac. Lobortis mattis aliquam faucibus purus in. Aliquet enim tortor at auctor urna nunc id cursus. Est pellentesque elit ullamcorper dignissim cras tincidunt. Lobortis feugiat vivamus at augue eget arcu dictum varius duis. Augue mauris augue neque gravida in fermentum. Massa placerat duis ultricies lacus sed turpis tincidunt id aliquet. Vitae semper quis lectus nulla at volutpat diam. Enim facilisis gravida neque convallis. Tempor orci eu lobortis elementum nibh tellus molestie.

Sed pulvinar proin gravida hendrerit lectus a. Diam vel quam elementum pulvinar etiam. Nulla posuere sollicitudin aliquam ultrices sagittis. Luctus venenatis lectus magna fringilla urna porttitor rhoncus. Vel orci porta non pulvinar neque laoreet suspendisse. In fermentum posuere urna nec. Venenatis tellus in metus vulputate. Aliquet lectus proin nibh nisl condimentum. Id consectetur purus ut faucibus pulvinar elementum integer. Ultrices sagittis orci a scelerisque purus. Purus gravida quis blandit turpis cursus. Euismod in pellentesque massa placerat duis ultricies. Justo nec ultrices dui sapien eget. Viverra mauris in aliquam sem fringilla ut morbi. Lacus suspendisse faucibus interdum posuere lorem ipsum.

Leo integer malesuada nunc vel risus. Dictum at tempor commodo ullamcorper a. Nulla facilisi cras fermentum odio eu feugiat. Consequat id porta nibh venenatis cras sed. Lectus nulla at volutpat diam ut venenatis tellus. Aliquam purus sit amet luctus venenatis lectus magna. In vitae turpis massa sed. Quis varius quam quisque id diam vel. Nulla posuere sollicitudin aliquam ultrices sagittis orci a scelerisque purus. Ullamcorper velit sed ullamcorper morbi tincidunt. Pharetra magna ac placerat vestibulum.</data>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_TO</name>
				<value>
					<var ref="to"/>
				</value>
			</staticParameter>
			<staticParameter>
				<name>_FROM</name>
				<value>
					<var ref="from"/>
				</value>
			</staticParameter>
		</staticParameters>
	</sharedLibrary>
</action>

Dear Peter,
Thank you so much for your details, after several attempts I am unable to make it run :slight_smile: , do you know if the password must be specified in base64 in the file?

Thank you!! :wink:

The password should be in clear text in the password file, no encoding is supported (yet).

Worth noting that in my example XML the password file is referenced via a variable - perhaps just use an absolute path to test, e.g. /tmp/passwd.txt, and see if it any better.

With v1.2+ you can now incorporate encoded passwords (and any other credentials) in the various configuration files. This looks like this:

password: ${enc:~/.keyfile:+encs+abc1234...}

Here, the keyfile is the same as used by the Gateway for secure passwords and the +encs+... string is also the same as used by the Gateway. By default, the Gateway uses a built-in key which you cannot access. You can use the new geneos aes ... commands to create both keyfiles and encode credentials or you can use the examples in the Gateway documentation.

Because credentials need to be symmetrically encoded - i.e. they have to be decodable to send to external systems - this is not strictly “secure” but it does protect against casual viewing (‘opaquing’) and also against the config file(s) being shared without the keyfile used to encode the credentials.

Thanks to real user feedback, even if it was internal, a bug/missing feature has been addressed and there will be a v1.2.1 branch, prob a beta tomorrow, before a v1.2.x branch final release.

Just to push this up the topic lilst, we have releases v.1.4.1. Please see the first post in this topic for more information!