X-Road

• 101
• Attachment(s) / Manused
• Errors
• Force restore conf backup
• Increasing open files limit
• Install specific version of xroad
• Links
• Messagelog Database info
• MISP2
• Op-mon to use one IP and daemon within a cluster
• REST examples
• rm -rf
• Testpäring
• Timeouts
• Tips & Tricks
• Upgrade procedure
• WSDL validator
• X-tee kataloog

101

ee-test/GOV/70007446/hksos/ambulanceResourcesV2/v1

keskkond/asutuse-tüüp/reg-kood/alamsüsteem/teenuse-nimi/teenuse-ver

Attachment(s) / Manused

soapi manust pole kunagi salvestatud,ainult soap body, või rest body

soap manus ei ela mitte kusagil seda ei salvestata, ja pole kunagi salvestatud. See on by design nii ,et soapi manusest jääb vaid hash signatuuri sees 🙂

kui turvik saadab sõnumi, siis võib olla korraks paneb tmp kausta, ja siis suunab kasutajale edasi ja see manus kustub igavesti

Errors

Server.ClientProxy.SslAuthenticationFailed
Service provider did not send correct authentication certificate

Klient annab teada et teenuse turvikuga ei õnnestu handshake teha sest ei tule teenusepakkuja poolt auth serti

Server.ServerProxy.ServiceFailed.HttpError>
Server responded with error 403: Forbidden

Võrgumure, nt LB ei luba päringul teenuse poole pöörduda

Server.ServerProxy.ServiceFailed.MissingHeaderField
Required field protocolVersion is missing

teenus tagastab päringu vastuse ilma protocolVersion elemendita

java.lang.OutOfMemoryError: Java heap space

https://confluence.niis.org/pages/viewpage.action?pageId=4292877

Error 403 Forbidden

Teie turvaserverile ei ole ajatempliteenust avatud ehk kas puudub leping või ei ole teenusepakkuja lubanud Teie turvaserveri IP-d ehk puudub teenusele ligipääs.

Error 503

HTTP veakood 503 tähendab, et teenus ei ole kättesaadav.

Server.ServerProxy.ServiceFailed.InvalidContentType: Invalid content type: text/html

Teenus vastas turvaserverile mingi sodiga, st mitte xml. Teenusega on midagi valesti

Server.ServerProxy.ServiceFailed.HttpError, Server responded with error 404: Not Found

Tähendab, et teenus on maas, mingi vale seadistus teenuses, pole õige url äkki.

Server.ServerProxy.ServiceFailed.IOError: Read timed out

Teenus annab timeouti, st teenus ei vasta turvikule

Server responded with error 502: Bad Gateway

soap:Server Internal error

Server.ServerProxy.ServiceFailed.HttpError Server responded with error 302

teenuse endpoint url vale, redirectib valesse kohta LB-st. url vajab vb / lõppu

Force restore conf backup

cat /etc/xroad/gpghome/openpgp-revocs.d/<-SOME-NUMBERS->.rev | grep uid

/usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome <-UID->

/usr/share/xroad/scripts/generate_gpg_keypair.sh /etc/xroad/gpghome <-UID->
su xroad
/usr/share/xroad/scripts/restore_xroad_proxy_configuration.sh -f /tmp/ss-automatic-backup-2022_03_08_031501.gpg -F -N

Increasing open files limit

Increasing xroad-proxy open file limits w/o host restart (temporary solution)

Increasing xroad-proxy open file limits

Install specific version of xroad

List available versions and pick Your poison

apt policy xroad-securityserver 2>/dev/null

Export version as env, for ex

export CANDIDATE=7.3.2-1.ubuntu22.04

Install

apt-get install xroad-addon-hwtokens="$CANDIDATE" \
xroad-addon-messagelog="$CANDIDATE" \
xroad-addon-metaservices="$CANDIDATE" \
xroad-addon-opmonitoring="$CANDIDATE" \
xroad-addon-proxymonitor="$CANDIDATE" \
xroad-addon-wsdlvalidator="$CANDIDATE" \
xroad-base="$CANDIDATE" \
xroad-confclient="$CANDIDATE" \
xroad-database-local="$CANDIDATE" \
xroad-monitor="$CANDIDATE" \
xroad-opmonitor="$CANDIDATE" \
xroad-proxy="$CANDIDATE" \
xroad-proxy-ui-api="$CANDIDATE" \
xroad-securityserver="$CANDIDATE" \
xroad-securityserver-ee="$CANDIDATE" \
xroad-signer="$CANDIDATE"

Links

X-tee kataloog - https://x-tee.ee/catalogue

Info - https://abi.ria.ee

NIIS Github - https://github.com/nordic-institute/X-Road

RIA Github X-Road scripts - https://github.com/ria-ee/X-Road-scripts

Päringuid testimiseks - https://abi.ria.ee/xtee/et/turvaserveri-haldus/soap-paeringud-testimiseks

Epoch timestamp converter - https://www.unixtimestamp.com

SK test certificate upload - https://demo.sk.ee/upload_cert/index.php

Messagelog Database info

https://moodle.ria.ee/mod/page/view.php?id=694

psql -h 127.0.0.1 -U messagelog <password from /etc/xroad/db.properties>

Kui palju kirjeid on tabelis

select count( *) from logrecord;

Milline on kõige vanem sõnumilogi kirje baasis

select to_timestamp(min( time )::float/1000) from logrecord;

Kui palju on ajatembedamata sõnumeid

select count(1) from logrecord where discriminator::text = 'm'::text and signaturehash is not null;

Viimane ajatembeldamata sõnum

select to_timestamp(min(time)::float/1000) from logrecord where discriminator::text = 'm'::text and signaturehash is not null;

Kui palju on ajatembeldatud aga arhiveerimata sõnumeid

select count(1) from logrecord where timestamprecord in (select id from logrecord where discriminator::text = 't'::text and archived = false);

Viimane ajatembeldatud aga arhiveerimata sõnum

select to_timestamp(min(time)::float/1000) from logrecord where timestamprecord in (select id from logrecord where discriminator::text = 't'::text and archived = false);

Size of the largest row in logrecord table

select t.id, t.archived, (pg_column_size(t.message)) as size from logrecord t where message IS NOT NULL order by size desc;

MISP2

If base package is upgraded, port 8080 conf will be defaulted

Uncomment port 8080 in /var/lib/tomcat8/conf/server.xml

Remote database upgrade

psql -h <database-ip> -p <database-port> -U postgres -d misp2db -f insert_xslt.sql

When was MISP2 upgraded to newer version

ls -t /var/log/dpkg* | xargs zgrep "upgrade xtee-misp2-application"

Admintool needs to be run as admin

/usr/xtee/app/admintool.sh

Conf

/var/lib/tomcat8/webapps/misp2/WEB-INF/classes/config.cfg

java.lang.OutOfMemoryError: Java heap space - increasing memory

Open /etc/default/tomcat8 and increase MaxPermSize=

misp apteegikood tuleb rets teenusest. add/edit = tervisekassa
Apteekrikoodi registreerimiseks peab pöörduma Tervisekassa poole, kes haldab 'rets' andmekogu teenuseid.

Op-mon to use one IP and daemon within a cluster

1) make sure, slave nodes -> master node can connect over port 2080

2) on master node, add to /etc/xroad/conf.d/local.ini

[op-monitor]
host = <master-internal-IP>

REST examples

listMethods

curl -X GET -H "accept: application/json" -H "X-Road-Client: ee-dev/GOV/70008799/pohak" "http://10.0.13.90/r1/ee-dev/GOV/70008799/pohak/listMethods" | json_pp 

HTTPS with cert and key

curl --cert nextcloud.bgp.12.berylia.org_cert.crt --key nextcloud.bgp.12.berylia.org_key.crt -X GET -H "accept: application/json" -H "X-Road-Client:BERYLIA/GOV/1003/recon" "https://xroad-securityserver.bgp.12.berylia.org/r1/BERYLIA/GOV/1002/satellite/listMethods" | json_pp

getOpenAPI

curl -k -X GET -H "accept: application/json" -H "X-Road-Client: ee-dev/GOV/70008799/pohak" "https://10.0.13.90/r1/ee-dev/GOV/70008799/pohak/getOpenAPI?serviceCode=adverse-event" | json_pp

POST

curl -k -X POST "https://10.0.14.26/r1/ee-test/GOV/70001969/tookeskk/digitaalne_teatis/api/v1/digitaalne_teatis" -H "accept: application/json" -H "Content-Type: application/json" -H "X-Road-Client: ee-test/GOV/70001969/tookeskk" \
--data '{
    "noticeId": "",
    "employeeIdCode": "",
    "employeeFirstName": "",
    "employeeLastName": "",
    "employeePhone": ""
}'

More help: https://x-tee.ee/docs/live/xroad/pr-rest_x-road_message_protocol_for_rest.html#64-post-request-and-response

rm -rf

apt purge --remove -y xroad-addon-hwtokens xroad-addon-messagelog   xroad-addon-metaservices xroad-addon-opmonitoring xroad-addon-proxymonitor xroad-addon-wsdlvalidator xroad-base               xroad-confclient         xroad-database-local     xroad-monitor            xroad-opmonitor          xroad-proxy              xroad-proxy-ui-api       xroad-securityserver-ee xroad-securityserver  xroad-signer

apt purge --remove -y postgresql postgresql-10 postgresql-client-10 postgresql-client-common postgresql-common postgresql-contrib

apt purge --remove -y rsyslog auditd && rm -rf /etc/rsyslog.d/* && rm -rf /etc/audit/rules.d/* 

apt autoclean -y && apt autoremove -y

userdel -r xroad
userdel -r xroad-slave
rm -rf /etc/xroad /var/lib/xroad /etc/xroad /etc/xroad.properties /usr/share/xroad /var/lib/xroad /var/log/xroad /var/tmp/xroad /etc/cron.d/xroad* /etc/systemd/system/xroad* /etc/zabbix/zabbix_agent2.d/userparameter_xroad* /etc/logrotate.d/xroad*

Testpäring

<?xml version="1.0" encoding="utf-8"?>
<SOAP-ENV:Envelope
        xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
        xmlns:xroad="http://x-road.eu/xsd/xroad.xsd"
        xmlns:id="http://x-road.eu/xsd/identifiers">
    <SOAP-ENV:Header>
        <xroad:client id:objectType="SUBSYSTEM">
            <id:xRoadInstance>EE</id:xRoadInstance>
            <id:memberClass>GOV</id:memberClass>
            <id:memberCode>70009770</id:memberCode>
            <id:subsystemCode>digilugu</id:subsystemCode>
        </xroad:client>
        <xroad:service id:objectType="SERVICE">
         <id:xRoadInstance>EE</id:xRoadInstance>
         <id:memberClass>GOV</id:memberClass>
         <id:memberCode>70009770</id:memberCode>
         <id:subsystemCode>digilugu</id:subsystemCode>
         <id:serviceCode>listMethods</id:serviceCode>
        </xroad:service>
        <xroad:id>{{v4uuid}}</xroad:id>
        <xroad:protocolVersion>4.0</xroad:protocolVersion>
    </SOAP-ENV:Header>
    <SOAP-ENV:Body>
        <xroad:listMethods/>
    </SOAP-ENV:Body>
</SOAP-ENV:Envelope>

curl -v -d @req.xml -o resp.xml -H "Content-type: text/xml; charset=UTF-8" http://TURVASERVER

Peale päringu tegemist piiluda resp.xml sisse, et näha päringu vastust

Timeouts

client-httpclient-timeout default=0
The maximum time (SO_TIMEOUT, in milliseconds) that connections from a service consuming security server to a service providing security server are allowed to wait for a response before the consumer end httpclient gives up. Value of 0 means that an infinite wait time is allowed

ma ütleks, et vaikimisi see polegi piiratud

kui turvikud loovad omavahel ühenduse, siis oodatakse seni, kuni üks pool loobub (timeouti pole). Vaikimisi producer ootab producer IS vastust 60sekundit, mida saab muuta producer admin UI-st

Tips & Tricks

When was xroad-securityserver package last upgraded

ls -t /var/log/dpkg* | xargs zgrep "upgrade xroad-securityserver"

Editing keyconf

1. On master node, stop xroad-signer process
2. Edit keyconf
3. On master node, start xroad-signer process
4. Restart xroad-signer on slave nodes

Grep packet loss

cat /var/log/xroad/*.log|grep -v "org.eclipse.jetty.io.EofException: null"|wc -l

Time opmonitor database read

time echo "select count(id) from operational_data;" | psql -h 127.0.0.1 -U opmonitor -W op-monitor

Keyconfist välja parseda serdid normaliseeritud kujul

cat /etc/xroad/signer/keyconf.xml | python -c "exec(\"import sys, xml.etree.ElementTree as ET\\nroot=ET.fromstring(sys.stdin.read())\\nfor cert in root.findall('.//cert/contents'):\\n print '-----BEGIN CERTIFICATE-----\\\\n'+str('\\\\n').join([cert.text[i:i+64] for i in range(0, len(cert.text), 64)])+'\\\\n-----END CERTIFICATE-----'\")"

Logback location

/etc/xroad/conf.d/

Global conf location in ss

/etc/xroad/globalconf/<environment>/shared-params.xml

List xroad services status

systemctl list-units "xroad*"

Stop all xroad processes

systemctl stop xroad-*

Environmental monitoring

is the monitoring of the X-Road environment: details of the security servers such as operating system, memory, disk space, CPU load, running processes and installed packages, etc.

Operational monitoring

is the monitoring of operational statistics such as which services have been called, how many times, what is the average response time, etc.

Count proxy open files

lsof -p `systemctl show -p MainPID xroad-proxy.service|cut -d'=' -f2`|wc -l

Upgrade procedure

https://www.x-tee.ee/docs/live/xroad/ig-xlb_x-road_external_load_balancer_installation_guide.html#72-online-rolling-upgrade

Enable maintenance mode(run from security server)

curl http://localhost:5566/maintenance?targetState=true

Watch connections and wait until all requests are finished

watch -n1 ss -tn state established sport = :5500 or dport = :5500

Make snapshot of machine

Unhold xroad packages

apt-mark unhold xroad-*

Update package list

apt update

Upgrade xroad packages

apt install xroad-securityserver-ee

Hold xroad packages

apt-mark hold xroad-*

Disable maintanance mode(run from security server)

curl http://localhost:5566/maintenance?targetState=false

WSDL validator

From Security Server CLI

/usr/share/xroad/wsdlvalidator/bin/wsdlvalidator_wrapper.sh tam6.wsdl

/usr/share/xroad/wsdlvalidator/bin/wsdlvalidator_wrapper.sh http://10.13.24.14/adapter/tam6.wsdl

Security Server uses Apache CXF wsdlvalidator

https://github.com/nordic-institute/X-Road/blob/develop/src/addons/wsdlvalidator/build.gradle#L22

https://cxf.apache.org/docs/wsdlvalidator.html

https://github.com/nordic-institute/wsdlvalidator

X-tee kataloog

Viga OpenAPI kirjelduse laadimisel või töötlemisel

Kontrollida üle "content-type HTTP header". Peaks vastama openapi kirjelduse formaadile, kas siis json või yaml, nt “application/json” või “application/yaml”

Kataloogi uuendatakse iga 1h tagant